# Ram Ready Digital Literacy Pilot Feedback Protocol

## Purpose

This protocol supports a small library-based usability and learning pilot with students, librarians, faculty, staff, accessibility reviewers, and community testers. It is designed to identify technical failures, confusing curriculum, missing supports, feature requests, and perceived usefulness before wider distribution.

The protocol adapts the qualitative course-review framework previously used to analyze University of Waterloo course reviews. It does not treat one rating or one comment as a final verdict. Feedback should be reviewed across recurring themes, specific pages, devices, and reviewer roles.

## Review dimensions

The website feedback form captures six core course-review dimensions:

1. **Organization and support** — Is the course clear, navigable, and supportive?
2. **Challenge, pacing, and learner agency** — Is the learning load appropriate, and can the learner make meaningful decisions?
3. **Assessment fairness and alignment** — Do choices and knowledge checks match the lesson objectives and explanations?
4. **Usefulness and transfer** — Can learners apply the skill in college, work, or daily life?
5. **Interest, fit, and expectations** — Are the examples relevant enough to sustain attention, and does the course deliver what it promises?
6. **Participation and modality** — Do the interactive format, mobile presentation, charts, and visual explanations support participation?

The form also measures:

- Visual usefulness
- Accessibility
- Source trust and verification
- Technical reliability
- Overall usefulness
- Recommendation likelihood
- Perceived confidence before and after testing
- Open-ended strengths, confusion, missing content, feature suggestions, and bugs

## Pilot size

A first library pilot can begin with approximately:

- 5–10 students
- 1–3 librarians or student-support staff
- 1–3 faculty, accessibility, or technology reviewers

This is a formative product test, not a representative research sample or a claim of institutional effectiveness.

## Suggested testing assignments

Distribute reviewers across different tasks so the pilot does not test only the landing page:

- Onboarding and guest story
- Foundations Episodes 1–5
- Foundations Episodes 6–10
- Foundations Episodes 11–15
- Foundations Episodes 16–20
- AI Quests 1–5
- AI Quests 6–10
- AI Quests 11–15
- AI Quests 16–20
- My Journey, achievements, progress, certificates, and Financial Futures handoff
- Mobile menu, small-screen layout, keyboard access, reduced motion, and dark mode

## Privacy

Do not request or record:

- Passwords or MFA codes
- Student ID numbers
- Grades
- Transcripts
- Private documents
- Health information
- Financial information
- Exact account information

The static feedback page stores no response. It prepares an email to `lmcgaffie@angelo.edu`, provides a copy option, and provides a JSON download. Testers decide whether to send the email.

## Analysis workflow

### Step 1 — Preserve the evidence

Save each returned feedback email or JSON file with a de-identified pilot ID. Keep the exact wording of open comments. Do not “clean up” comments before coding.

### Step 2 — Separate technical and curriculum findings

Classify each issue as one or more of:

- Blocking technical failure
- Non-blocking usability problem
- Accessibility problem
- Content accuracy or source concern
- Clarity or organization problem
- Pacing or challenge concern
- Knowledge-check alignment concern
- Missing feature or support
- Positive usefulness or transfer evidence
- Interest or relevance concern

### Step 3 — Code recurring themes

Use the six review dimensions above as an initial codebook while allowing new themes to emerge. Do not force every comment into a preselected category.

### Step 4 — Connect themes to evidence

Maintain an evidence matrix containing:

- Pilot response ID
- Reviewer role
- Device and browser
- Page or lesson
- Exact comment excerpt
- Theme code
- Severity
- Recommended action
- Resolution status

### Step 5 — Prioritize changes

Use this order:

1. Privacy or security problem
2. Blocking navigation, menu, quiz, progress, or certificate failure
3. Accessibility barrier
4. Incorrect or unsupported instructional claim
5. Unresolved placeholder or broken personalization
6. Confusing interaction or assessment
7. Missing content or feature
8. Cosmetic improvement

### Step 6 — Validate fixes

After a change, reproduce the original tester steps on the same device class where possible. Mark the issue resolved only when the behavior is verified.

## Reporting

A pilot summary should report:

- Number and roles of testers
- Pages and lessons tested
- Devices and browsers represented
- Recurring strengths
- Recurring problems
- Blocking issues
- Accessibility findings
- Most requested features
- Changes completed
- Changes deferred
- Limitations of the pilot

Do not claim that the course is effective for all students based on a small convenience pilot.