Integration or Transformation?

A cross-national study of information and communication technology in school education


Appendix 6.7

6.7      Instruments for measuring ICT integration and results of meta-studies

Table 17: Some ways of measuring ICT integration in schools (USA).



Strands, attributes, or components measured using the instrument

Lemke & Coughlin, 1998


Milken Family Foundation

Seven Dimensions for Gauging Progress of Technology in the Schools


learning environments

professional competency

system capacity

community connections

technology capacity


Jones, Valdez, Nowakowski & Rasmussen, 1995



North Central Regional Educational Laboratory (NCREL)

Plugging In:

choosing and using educational technology

Engaged Learning:

vision of learning



instructional model

learning context


teacher roles

student roles

High Technology:





ease of use

functionality, 1998

Maturity Model Benchmarks Survey ver. 2.5






National Study of School Evaluation, 1998

Indicators of Quality Information Technology Systems in K-12 Schools

Student Learning:

information technology basics

application and integration of technology

use of creativity tools

technology in life and society

School Instructional Capacity:

curriculum development

instructional strategies and learning activities

assessment of student learning

School Organisation:

leadership, vision and planning

professional development



creating a community of learners

Bender, 2000

Measuring Technology Integration in Learning Environments

Teacher Response Instrument

Teacherís beliefs and intentions regarding technology use and creating an environment for learning

The Classroom Observation instrument consists of 21 indicators. Nineteen of these indicators are rated 0Ė3.the last two are Technology Used and Software Used which are simply lists of the items observed in use during the lesson.

(Bingham, 2000)

Table 18: Meta-analyses of ICT in education (Sinko & Lehtinen, 1999)


Studies analysed

(Positive effect sizes indicate computers were found to enhance learning)

Ryan, A. W. (1991) Meta-analysis of achievement effects of microcomputers applications in elementary schools. Educational Administration Quarterly, 27(2), 161-184.


The average effect size was 0.31.

The computer software used was classified into these categories:

        drill and practice programs



        programming language

        discovery programs


A combination of different forms of use was generally more effective than using computers in only one way.

Kulik, C-L. C. & Kulik J. A. (1991) Effectiveness of computer-based instruction. An updated analysis. Computers in Human Behavior, 7, 75-94.


This study also showed that cognitive achievement was improved by computer-aided instruction (effect size 0.30) . Later, Kulik (1994) published a summary of 12 previous meta-analyses, which turned out to have parallel results. The effect sizes found in these summaries reached all the way up to 0.50. Such large effect sizes already signify essential improvements in learning outcomes. (p. 37).

Liao, Y. C. & Bright, G. W. (1991) Effects of computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7 (3), 251-268.


Experimenters usually obtained higher effect sizes in short experiments and with computer environments which emphasise self-directed learning.  (p. 38)

Khaili, A. & Shashaani, L. (1994) The effectiveness of computer applications: a meta-analysis. Journal of Research on Computing in Education, 27, 48-61.


The mean effect size ... was 0.38. Found evidence of Hawthorne effects (the novelty wears off after a while) and group size effects (smaller groups got larger positive effects).

Fletcher-Flinn, C. M. & Gravatt, B. (1995) The Efficacy of Computer-Assisted Instruction (CAI): A Meta-Analysis. Journal of Educational Computing Research, 12, 19-242.


For the various subjects, the greatest effectiveness was obtained in the teaching of mathematics.

Table 18 (continued)


Studies analysed


Cotton, K. (1997) Computer-assisted instruction. Portland, OR: Northwest Regional Educational Laboratory. School Improvement Research Series.


The use of computers and word-processing software led to superior writing performance than traditional pencil-and-paper work, as manifested by longer written samples, greater variety of word usage, more variety of sentence structures, more substantial revision, greater responsiveness to teacher and peer feedback, better understanding of the writing process and better attitudes towards writing.

The use of computers in teaching also led to positive changes in attitudes towards school and learning in general, as well as in motivation.

The studies showed that using computers in instruction yielded different results for different student populations: low achievers and handicapped students benefited more than higher-achieving students; positive effects were greater with young students than with older ones; students with a weak socio-economic background benefited more than students whose parents were wealthy and highly educated; boys benefited more from computer-aided instruction than girls.

The positive effects of computer-aided drill and practice programs were especially visible with respect to simple cognitive tasks such as retaining learned material and doing various routine tasks. (p. 35-36)

Liao, Y-K. (1998) Effects of hypermedia versus traditional instruction on studentís achievement. Proceedings, annual meeting of American Educational Research Association, San Diego, CA.


In most cases hypermedia environments produced better learning outcomes than traditional instruction. ...The mean effect size was 0.48

Lehtinen, E., Hakkarainen, K., Lipponen, L., Rahikainen, M. & Muukkonen, H. (1998) Computer supported collaborative learning: A review of research and development. CL-Net. A report for the European Commission.


Where computers were formerly viewed as a tool for individualising instruction, they are now viewed more and more as a tool for social interaction and collaborative activities. Many studies report positive learning effects with computer-supported collaborative learning. (p. 44)