Description

Assignment

A student developed for her dissertation a 36-item scale for raters to evaluate the quality of teacher instruction by video.  To evaluate the measurement process, she had 5 teacher observation experts rate videotaped lessons from 3 different teachers. Each teacher submitted one 45-minute lesson for evaluation. The lessons differed across teachers.  Each of the 5 expert raters evaluated all 3 teacher-videos using all 36 items.  Items were scored on a 1-4 integer scale. 

a.      Write out the full model for the Generalizability Theory design that best represents this procedure. Include at least 1 or 2 examples of specifications of error distributions (or writing them all out is fine if it’s not too tedious).

Note:

·adapt the G theory equation to match the scenario

·write the full model (equation)

·the equation will definitely have a random error. Just specify that error with rationale.

b.      Of the facets of error whose variance components you cannot estimate, which do you think is of greatest concern?

Note:,

·         Items (i)

·         Raters (r)

·         Teachers (t)

·         What is the measurement design?

·         Nested or crossed with?

·         Draw a Venn diagram to show that , write a rationale

·         Then write a single sentence to answer the question clearly

c.       The student was interested in comparing the reliability of these 5 expert raters to principal raters. For principals, she used the same design but with 65 principals (each of the 65 principals rated all 3 teacher-videos, using all 36 items). She obtained the estimated variance components below for experts and for principals.  She used the same videos and items across both studies.

I provide D study curves for the same design on the next page for both rater types, experts and principals, with relative standard errors at left and generalizability coefficients for relative error at the right.  Based on the variance components above, identify whether the top two plots are for Experts or for Principals.  Conduct any calculations you like to support your decision, although an explanation may also suffice.

Note:,

·         There are differences between variance components of principals and raters.

·         Which of the diagrams below are for principals and which one if for experts. Make the decision based on the values.

·         Provide the reason

·         What are the X and what are Y in the diagrams.

·         Calculate and tell why?