Skaidrė 1 -

Skaidrė 1 -

Model and method for evaluation of eQNet travel well quality learning resources Assoc. Prof. Dr. Eugenijus Kurilovas, ITC, Lithuania EdReNe final strategic seminar, Barcelona, 26 March 2010 eQNet project LRE particularly seeks to identify LOs that travel well (i.e., reusable) across national borders and can be used in a cultural and linguistic context different from the one in which they were created eQNet is a three-year (September 2009-2012) Comenius Multilateral Network funded under the European Commissions Lifelong Learning programme. The project is coordinated by European Schoolnet and involves 9 Ministries of Education - AT, BE, CZ, IT, LT, NO, PT, SK, SE or agencies nominated to act of their behalf

The primary aim is to improve the quality of LOs in LRE. eQNet will do this by establishing a network consisting of policy makers, researchers, and practitioners (teachers) that will develop and apply travel well quality criteria to both existing LRE content as well as that to be selected in future from national repositories eQNet project The vision driving the LRE is that a significant percentage of high quality LOs developed in different countries, in different languages and to meet the needs of different curricula can be re-used at European level eQNet will provide a forum for joint reflection and co-operation related to the exchange and re-use of educational content and allow network members to: 1. better share information and expertise particularly related to

travel well quality criteria (pedagogical, technical and intellectual property rights (IPR) factors) 2. develop new frameworks (including a network of expert teachers) to improve the quality of LOs and metadata in both national repositories and the LRE, including the growing volume of user-generated content and metadata, as well as to improve the multilinguality of LRE content as a result of the translation of metadata, making use, where appropriate, of automatic metadata translation approaches and technologies 3. enable schools to participate in a community of practice related to the use LOs at European level eQNet project Major results will include: the development of travel well quality criteria to more easily identify LOs with the potential for cross-border use (this WP is coordinated by the Lithuanian MoE

represented by ITC, and EUN) the practical application by teachers of these criteria to >3,500 LOs in the LRE showcases of the best of these LOs in a travel well section of the LRE portal where necessary, the enrichment of selected LOs with new or better metadata a community of practice for teachers around these LOs Applied scientific methods and principles The aim of the presentation is to suggest scientific model and

method for expert evaluation of quality of LOs paying especial attention to LOs reusability level The main problem of all existing approaches in the area is high level of expert evaluation subjectivity Several scientific methods and principles to minimise subjectivity level in expert evaluation of LOs quality were applied, namely: 1. principles of multiple criteria decision analysis for identification of quality criteria 2. technological quality criteria classification principle 3. fuzzy group decision making theory to obtain evaluation measures 4. normalisation requirement for criteria weights 5. scalarisation method for LOs quality optimisation The complex application of these approaches could significantly improve the quality of expert evaluation of LOs and noticeably reduce the expert evaluation subjectivity level

An example of practical application of these approaches for LOs quality evaluation is also presented LOs quality evaluation model (criteria tree) Outline To present different approaches to evaluation of Learning Objects (LOs) previous works To present the principles of identification of quality evaluation criteria To present LOs technological quality evaluation model (criteria tree) To present overall LOs quality evaluation model (incl. technological, pedagogical and IPR criteria)

LOs quality evaluation model (criteria tree) The main notions: LO is referred here as any digital resource that can be reused to support learning (Wiley, 2000) Quality evaluation is defined as the systematic examination of the extent to which an entity (part, product, service or organisation) is capable of meeting specified requirements (ISO/IEC 14598-1:1999, 1999) Reusability of LOs (or their ability to travel well between different contexts and education systems) is considered here as a part of the overall quality of LOs This means that any high quality LO has some reusability level (or potential to travel well), but this

does not mean that any reusable LO is quality one LOs quality evaluation model (criteria tree) The need for reusability of LOs has at least three elements: Interoperability: LO is interoperable and can be used in different platforms Flexibility in terms of pedagogic situations: LO can fit into a variety of pedagogic situations Modifiability to suit a particular teachers or students needs: LO can be made more appropriate to a pedagogic situation by modifying it to suit a particular teachers or students needs (McCormick et al., 2004) LOs quality evaluation model (criteria tree) Learning Object Review Instrument (LORI):

Presentation: Aesthetics Presentation: Design for learning Accuracy of content Support for learning goals Motivation Interaction: Usability Interaction: Feedback and adaptation Reusability Metadata and interoperability compliance Accessibility LOs quality evaluation model (criteria tree) Leacock & Nesbit (2007) quality criteria:

Content quality: veracity, accuracy, balanced presentation of ideas, and appropriate level of detail) Learning goal alignment: alignment among learning goals, activities, assessments, and learner characteristics Feedback and adaptation: adaptive content or feedback driven by differential learner input or learner modelling Motivation: ability to motivate and interest an identified population of learners Presentation design: design of visual and auditory information for enhanced learning and efficient mental processing Interaction usability: ease of navigation, predictability of the user interface, and the quality of the interface help features Accessibility: design of controls and presentation formats to accommodate disabled and mobile learners Reusability: ability to use in varying learning contexts and with learners from different backgrounds Standards compliance: Adherence to international standards and

specifications LOs quality evaluation model (criteria tree) Becta (2007) core pedagogic principles: Inclusion and access Learner engagement Effective learning Assessment to support learning Robust summative assessment Innovative approaches Ease of use Match to the curriculum LOs quality evaluation model (criteria tree) Becta (2007) core design principles:

Digital learning resource design Robustness and support Human-computer interaction Quality of assets Accessibility Interoperability Testing and verification Effective communication LOs quality evaluation model (criteria tree) Paulsson and Naeve (2006) have suggested six areas for establishing LO technological quality criteria:

A narrow definition A mapping taxonomy More extensive standards Best practice for use of existing standards Architecture models The separation of pedagogy from the supporting technology of LOs LOs quality evaluation model (criteria tree) MELT set of the LOs quality criteria has been divided into five categories: Pedagogical Usability Reusability Accessibility

Production LOs quality evaluation model (criteria tree) MELT: the cross-border reuse of content will be more likely if LOs: Have a strong visual element and users can broadly understand what is the intended learning objective or topic. E.g., LOs may have little or no text; and include animations and simulations that are self-explanatory or have just a few text labels or icons / buttons for start, stop, etc. Have been designed to be language customisable and are already offered in more than one language Address curriculum topics that could be considered trans-national

Are adaptable from a technical (e.g., LOs are supplied along with an authoring environment or tools) or IPR perspective LOs quality evaluation model (criteria tree) LOs Q4R (Quality for Reuse) criteria: Strategies before LO inclusion in the LOR are based on the following principles: Only build or integrate LOs, which can be certified for quality Interactive LOs are software and as such should answer to software quality criteria Strategies during LO inclusion in the LOR are based on: The principle of reducing form-filling Use of guiding wizards, smart automatic and semi-automatic computer agents to assist in assuring technical

interoperability Strategies after LO inclusion in the LOR are based on the following principles: Provide interesting and easily understood user statistics, such as stars, percentages, voting systems Include recommendations for reuse by the user, both to the next user and the designer LOs quality evaluation model (criteria tree) Belgium (Flanders) nationally recognised LOs quality criteria: Scope Criteria: Aims and users (Aims; Target group) Contents (Curriculum area; Relevant source; Source type; Granularity; No advertisements) Geographical coverage (Geographical Limitations (availability relevance); Language) Accessibility (Technology; Costs; Registration)

Source data (Metadata) Quality criteria: Content criteria (Validity; Accuracy; Authority; Actuality; Substantial Coverage; Completeness) Formal criteria (Navigation; User support; Use of ICT and standards) Process criteria (Integrity of information; Site stability; Platform stability) LOs quality evaluation model (criteria tree) Czech LOs quality criteria: Content accuracy Connection with national curriculum Didactics / Pedagogy accuracy Typographic rules Citations, Copyright LOs quality evaluation model (criteria tree) French LOs quality criteria:

Curriculum conforming Free rights for pedagogical use in the classroom and out of the classroom Respect of publishers engagements Provide access to expertise the resources Update information must be available Provide the possibility to block advertising (if included) LOs quality evaluation model (criteria tree)

Lithuanian LOs quality criteria: Methodical aspects User interface (incl. personalisation) LOs arrangement possibilities Communication & collaboration possibilities and tools Technical features (incl. working stability) Documentation Implementation and maintenance expenditure LOs quality evaluation model (criteria tree) Norwegian LOs quality criteria: Technical interoperability (File formats, standards (IMS QTI & CP, etc.)) Metadata tagging Accessibility (WCAG 1.0) Parallel publishing (two forms of the Norwegian language exist)

Pedagogical issues LOs quality evaluation model (criteria tree) Portuguese LOs quality criteria: Technical Scientific Educational Language Values and Attitudes LOs quality evaluation model (criteria tree) Swedish LOs quality criteria: Every web site must have source information and must not conflict with Swedish law The presentation must be clear and easy to navigate

The information must be reliable: who is responsible for the information, why the information is published, how often it is updated, etc. Other criteria deal with general interest, language and access requirements LOs quality evaluation model (criteria tree) Swiss LOs quality criteria: National level: Recommendation for description (Application profile) Technical tools Harvesting Local / Partner level: Selection and validation of resources according local criteria and culture

Metadata check, enrichment and maintenance Publication (local) LOs quality evaluation model (criteria tree) What is important: Alternatives (LOs to be chosen) Principles of identification of quality evaluation criteria Stakeholders LOs quality evaluation model (criteria tree) The main problem here is how to establish the proper set of LOs evaluation criteria which should reflect objective scientific principles of construction a model (criteria tree) for LOs quality evaluation These general principles have been analysed in several multiple criteria decision analysis (MCDA) related works such as (Belton, Stewart, 2002)

If the set of decision alternatives is assumed to be predefined, fixed and finite, then the decision problem is to choose the optimal alternative or, maybe, to rank them. But usually the experts (decision makers) have to deal with the problem of optimal decision in the multiple criteria situation where the objectives are often conflicting. In this case, an optimal decision is the one that maximises the decision makers utility. An initial candidate set of key factors, which we term criteria should emerge from the problem structuring process In identifying these, the following considerations are relevant to all multiple criteria decision analysis approaches: LOs quality evaluation model (criteria tree) Value relevance: Are the decision makers able to link the

concept to their goals, thereby enabling them to specify preferences which relate directly to the concept? Understandability: It is important that decision makers have a shared understanding of concepts to be used in an analysis. Measurability: All MCDA implies some degree of measurement of the performance of alternatives against specified criteria, thus it must be possible to specify this in a consistent manner. It is usual to decompose criteria to a level of detail which allows this. Non-redundancy: Is there more than one criterion measuring the same factor? When eliciting ideas often the same concept may arise under different headings One can easily check for criteria which appear to be measuring the same thing by calculating a correlation coefficient if appropriate data is available, or carrying out a process of matching as associated with analysis of repertory grids.

LOs quality evaluation model (criteria tree) Judgmental independence: Criteria are not judgementally independent if preferences with respect to a single criterion, or trade-offs between two criteria, depend on the level of another. Balancing completeness and conciseness: a number of authors note that desirable characteristics of a value tree are that it is complete, i.e., that all important aspects of the problem are captured, and also that it is concise, keeping the level of detail to the minimum required. Operationality: The model is usable with reasonable effort that the information required does not place excessive demands on the decision makers. The context in which the model is being used is clearly important in judging the usability of a model.

Simplicity versus complexity: The value tree, or criteria set is itself a simple representation, capturing the essence of a problem, which has been extracted from a complex problem description. The modeller should strive for the simplest tree which adequately captures the problem for the decision maker. LOs quality evaluation model (technical criteria tree) LOs quality evaluation model (criteria tree) Software engineering Principle: we can evaluate the software using the two types of evaluation criteria internal quality and quality in use criteria: Internal quality is a descriptive characteristic that describes the quality of software independently from any particular context of its use (decision makers scientists, experts in informatics engineering) Quality in use is evaluative characteristic

of software obtained by making a judgment based on criteria that determine the worthiness of software for a particular project or user (decision makers users) has identified some travel well parameters: LOsEUN quality evaluation model (criteria tree) Language independence / Low language dependence (easily translatable) or Multilinguality Strong visual structure (animations, images and short videos are travelling best) No large semantic density (easy to include in any curriculum)

Other parameters more relating to quality in general: Cultural appropriateness: the resource respects the cultural and social norms of teacher and / or pupil Usability: users can find their way through the resource almost intuitively. This may be possible even if the user is unfamiliar with the language of the resource Technical features: easy to access (no login or plugins) Openness: open license, free to use, open code (like LOs quality evaluation model (criteria tree) LOs quality evaluation method Outline To present the ratings (values) of LOs evaluation criteria (taking into account technological, pedagogical and

intellectual property rights (IPR) criteria) To present the method for evaluation of LOs (experts additive utility function) To provide an example of practical evaluation of three LOs alternatives using this method LOs quality evaluation method Expert evaluation is referred here as the multiple criteria evaluation of the learning software (such as LOs) aimed at selection of the best alternative based on score-ranking results. Learning software multiple criteria evaluation method is referred here as the experts additive utility function including the alternatives (i.e., different LOs) evaluation criteria, their ratings (values) and weights.

This method is well-known in the theory of optimisation methods and is named scalarization method. LOs quality evaluation method m f ( X ) ai f i ( X ) m i 1 ai 1 i 1 , , ai 0 where X

LO alternative i = {1, , m} evaluation criteria (m=8) rating (value) of criterion fi ( X ) normalized weights of ai 0 criteria LOs quality evaluation method The widely used measurement criteria of the decision attributes quality are mainly qualitative and subjective. Decisions in this context are often expressed in natural language, and evaluators are unable to assign exact numerical values to the different criteria. Assessment can be often performed by linguistic variables: bad, poor, fair, good and excellent. These values are imprecise and uncertain: they are commonly called fuzzy values. Integrating these different judgments to obtain a final evaluation is not evident. Therefore, (Ounaies et al., 2009) have proposed to use fuzzy group decision making theory to obtain final assessment measures. Linguistic variables conversion into non-fuzzy values:

Excellent Good Fair Poor 0.850 0.675 0.500 0.325 LOs quality evaluation method Value Bad (0.150) Explanation Failed or feature (criterion) does not exist Poor (0.325) Has poor support and / or it can be done but with significant effort

Fair (0.500) Fair support but needs modification to reach the desired level of support Good (0.675) Good support and needs a minimal amount of effort Excellent (0.850) Excellent support and meets the criteria out of the box, minimal effort LOs quality evaluation method Examples of Lithuanian LOs (Physics): LO1: Light diffraction (viesos difrakcija) LO2: Photo effect (Fotoefektas)

LO3: Isobar process (Izobarinis procesas) 395/ LOs quality evaluation method (equal weights 0.125) Evaluation criteria LO1 LO2 LO3 Technological quality criteria Technological reusability 0.850 0.850 0.850 Design and usability 0.850 0.850

0.850 Working stability 0.850 0.850 0.850 Architecture 0.675 0.675 0.675 Multimedia interactivity level 0.500 0.500 0.500

Language independence 0.850 0.675 0.500 Ease of use, intuitiveness 0.850 0.850 0.850 0.850 0.850 0.850 0.7844

0.7625 0.7406 Pedagogical quality criteria IPR criterion Open licence, cost Evaluation results (values * 0.125) LOs quality evaluation method (weights 0.2; 0.1) LO1 LO2 LO3 Technological reusability (weight 0.2) 0.1700 0.1700

0.1700 Design and usability (weight 0.1) 0.0850 0.0850 0.0850 Working stability (weight 0.1) 0.0850 0.0850 0.0850 Architecture (weight 0.1) 0.0675 0.0675 0.0675

Multimedia interactivity level (weight 0.1) 0.0500 0.0500 0.0500 Language independence (weight 0.2) 0.1700 0.1350 0.1000 Ease of use, intuitiveness (weight 0.1) 0.0850 0.0850

0.0850 0.0850 0.0850 0.0850 Evaluation criteria Technological quality criteria Pedagogical quality criteria IPR criterion Open licence, cost (weight 0.1) LOs quality evaluation method Table 1 results mean that LO1 meets 78.44% quality in comparison with the ideal, LO2 76.25%, and LO3 74.06%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from overall quality point of view.

Table 2 results mean that LO1 meets 79.75% travel well quality in comparison with the ideal, LO2 76.25%, and LO3 72.75%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from travel well quality point of view. References ISO/IEC 14598-1:1999 (1999). Information Technology Software Product Evaluation Part 1: General Overview. First edition, 1999-04-15 Becta (2007). Quality principles for digital learning resources. Available online at section=re&catcode=ss_res_dig_02&rid=13293 Kurilovas, E.; Dagien, V. (2009). Multiple Criteria Comparative Evaluation

of e-Learning Systems and Components. Informatica. 2009, Vol. 20, No. 4, pp. 499-518 [ISI Web of Science] Kurilovas, E.; Dagien, V. (2009). Quality Evaluation and Optimisation of e-Learning System Components. In: Proceedings of the 8th European Conference on e-Learning (ECEL09). Bari, Italy, October 2930, 2009, pp. 315324 [ISI Web of Knowledge] [ERIC] the best paper award Kurilovas, E., Dagiene, V. (2008). Learning Objects and Virtual Learning Environments Technical Evaluation Tools. In: Proceedings of the 7th European Conference on e-Learning (ECEL08). Agia Napa, Cyprus, 67 November 2008, Vol. 2, pp. 24 33 [ISI Web of Knowledge] [ERIC] the best paper award References Leacock, T. L. & Nesbit, J. C. (2007). A Framework for Evaluating the Quality of Multimedia Learning Resources. Educational Technology & Society, 10 (2), pp. 4459

McCormick, R., Scrimshaw, P., Li, N., and Clifford, C. (2004). CELEBRATE Evaluation report. file/Deliverable7_2EvaluationReport02Dec04.pdf MELT (2008). EU eContentplus programmes MELT (Metadata Ecology for Learning and Teaching) project web site. Available: Panzavolta, S. (2009). Travel-well resources. Presentation in the EMINENT conference. Vilnius, 26-27 December, 2009 Paulsson, F., Naeve, A. (2006). Establishing technical quality criteria for Learning Objects. Available: References

Ounaies, H.Z.; Jamoussi, Y.; Ben Ghezala, H.H. (2009). Evaluation framework based on fuzzy measured method in adaptive learning system. Themes in Science and Technology Education. Vol. 1, Nr. 1, 2009, pp. 4958 Q4R (2008). Quality for Reuse (Q4R) project web site. Available: Van Assche, F., and Vuorikari, R. (2006). A Framework for Quality of Learning Resources. In: European Handbook for Quality, CEDEFOP, 2006 Vargo, J., Nesbit, J. C., Belfer, K., Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and interrater reliability. International Journal of Computers and Applications, Vol. 25 (3), pp 198205

Wiley. D. A. (2000). Connecting Learning Objects to Instructional design Theory: a definition, a Metaphor, and a Taxonomy. Utah State University. Thank you for your attention !!! Questions ? Dr. Eugenijus Kurilovas [email protected]

Recently Viewed Presentations

  • The Biological Basis for Behavior

    The Biological Basis for Behavior

    Proposed by Hobson and McCarley in 1977 it is a biological theory of dreams. b. It hypothesizes that dreams are random firings of the brain that your brain tries to make sense of by creating a story - essential dreams...
  • Bau Transport og engros årskonference 2019

    Bau Transport og engros årskonference 2019

    Arbejdsgiveren skal herudover sørge for at føre effektivt tilsyn med, at den ansatte rent faktisk udfører arbejdet sikkerheds- og sundhedsmæssigt forsvarligt og følger instruktionerne.Arbejdsgiveren kan vælge at lade en anden udføre opgaven med oplæring, instruktion og tilsyn på sine vegne.


    national holidays and traditions in the usa
  • Words, words, words You say them all the

    Words, words, words You say them all the

    This is unlikely due to what we know of its origin. The sunburn or pellagra explanation seems more likely than the anger one. Interestingly, the Afrikaans Rooinek, which literally means redneck, is a disparaging term the Boers used to apply...
  • Alaska Airlines Center Revenues

    Alaska Airlines Center Revenues

    Alaska Airlines Center:Revenues & Budgeting for a new Arena. Garrett James, Michael Batch, Tim Mollerstrom - PADM 628 Public Financial Management. Rationale for Creation. Current UAA facility Wells Fargo Sports Complex (WFSC) is no longer a sufficient facility for multiple...
  • The mole unit o What is 1 mole

    The mole unit o What is 1 mole

    sp gr 60o 60o - 131.5 EXAMPLE (mass fr. , mole fr and Average MWt ) What is the avg MWt of a gas mixture that contains: Mass, Kg H2 3 CH4 8 C2H4 56 Total 67 In the Petroleum...
  • Building a Thesis & Works Cited Page

    Building a Thesis & Works Cited Page

    Too Use too as a synonym for also or to indicate excessiveness before a verb. ... (they are) going to the movies. Their possessive Their commitment is solid and unwavering. There, their, or they're Title: Building a Thesis & Works...
  • Canadian Native Art Native Art in Canada Canadian

    Canadian Native Art Native Art in Canada Canadian

    The Blue Raven (1989) is the story of a heroic journey a Native boy makes to find help for his suffering people. Harrison has also illustrated Robert Service's The Cremation of Sam McGee (1986) and The Shooting of Dan McGrew...