As AI-powered learning tools become increasingly prevalent, safeguarding should be considered throughout the life of a product—from initial design and development through deployment and ongoing improvement. As AI capabilities evolve, developers must constantly anticipate and address potential risks to learners, educators, and other users.
For product teams looking for practical guidance on how to build and implement AI-enabled tools responsibly, the Safeguarding Guide for the Safe and Responsible Use of AI in Ed Tech offers a useful resource. It outlines six principles: focus on educational outcomes; direct generative AI systems to support students’ learning and wellbeing; ensure the privacy and security of user data; prioritize accessibility and fairness; promote transparency and explainability; and give users control over their data.
At an event hosted by our team, Cristina Heffernan from ASSISTments and Agustin Pardo Van Thienen from Wumbox shared how safeguarding shapes their day-to-day product decisions. Although their products serve different age groups and use cases, the safeguarding challenges they face and the solutions they are building illustrate many of the principles and related practices outlined in the guide.
This post explores the safeguarding principles and practices discussed during the event, offering concrete examples of how product teams can translate principles into everyday design and implementation decisions.
Designing With Safety in Mind From the Start
One of the earliest and most consequential safeguarding decisions is what data a tool actually needs to exist in the first place.
Teams should think carefully about what data is truly required versus what is simply convenient to collect. A consistent principle that emerged from the discussion is data minimization — collecting only what is necessary to deliver learning value, personalization, and functionality.
In practice, this looks different depending on the product context. For example, ASSISTments is designed so that students provide only basic login information that enables teachers to access reports, while everything else centers on the learning activity. Wumbox, however, supports neurodiverse learners, and requires additional information related to learning differences in order to personalize learning experiences. In each case, the underlying design constraint remains the same: collect intentionally, not by default, and continuously evaluate whether each data element is actually necessary.
A key takeaway for teams is that safeguarding risk is often introduced long before technology is involved. Every additional data field increases system complexity and expands the potential for unintended exposure, misuse, or retention challenges.
Separating Personally Identifiable Information (PII) from Data
Once data begins flowing through a tool, another important design decision emerges: how tightly personally identifiable information (PII)–data that can identify a learner such as names or email addresses–is coupled to learning data.
Teams should consider whether their specific use cases — personalization, classroom insights, research, etc. — actually require persistent linkage between PII and behavior. In many cases, performance data, response patterns, and engagement signals can and should be analyzed independently of direct identifiers while still supporting adaptive learning experiences.
This separation becomes especially important as AI systems are introduced into the learning pipeline. Platforms can adjust difficulty, recommend next steps, or surface classroom-level trends using aggregated interaction data rather than relying on knowledge of PII. In many cases, the system does not need to “know” who the learner is in order to function effectively.
For product teams, this is not only a privacy safeguard but also an architectural advantage. Decoupling PII enables more flexible analytics, supports research partnerships under controlled conditions, and reduces risk while preserving core product functionality.
When Inputs Get Messy: Open Responses and Media
Safeguarding complexity increases dramatically when moving beyond structured inputs, such as multiple choice or fixed responses.
Teams should think carefully about how open-ended responses and multimodal inputs are handled, as these introduce less predictable and often higher-risk data. For example, a student may include personal details in a written explanation, or an image of handwritten work might capture faces, names, or home environments that are not relevant to the learning task itself.
Because of this, user-generated content may require different handling approaches. In some cases, open-response text and image-based inputs are excluded from broader or publicly shared datasets due to their higher sensitivity. When they are included, they must be managed with stricter policies, additional anonymization steps, and more controlled access patterns.
For product teams, this reinforces a key principle: safeguarding strategies should be tailored to specific contexts. While strong protections should apply across the board, higher-risk data may warrant additional controls, storage considerations, or review layers.
Where Automation Stops: Keeping Humans in the Loop
As generative AI becomes more deeply embedded in learning products, human oversight remains a critical safeguard.
Teams should think carefully about where AI is accelerating workflows versus where it is making decisions that directly affect learners. Human review should be used as a checkpoint before AI-generated instructional materials, practice problems, or feedback are released to students. This includes cases where AI is used to generate new learning content, with human review ensuring that what reaches classrooms meets appropriate quality and safety standards. While this may slow deployment slightly, it significantly reduces the risk of incorrect or inappropriate content reaching learners.
Relatedly, even as cloud infrastructure and third-party systems provide baseline security capabilities, responsibility for safe deployment still sits with product teams — including how systems are configured, monitored, and audited in practice.
Across contexts, the underlying principle is consistent: AI should not replace responsibility. Human oversight remains one of the most reliable safeguards in learner-facing systems.
Transparency as a Product Feature
Safeguarding is not only about backend systems. It also shows up in how products communicate with users.
When AI is used to generate feedback or instructional support, learners should be made aware of this. In practice, this can include explicitly labeling AI-generated responses so students know when feedback is produced by an automated system rather than a human instructor. This helps ensure that AI outputs are not treated as unquestioned authority and supports more appropriate use in learning contexts.
Transparency also extends to broader data practices. Product teams should clearly communicate what data is collected, how it is used, how long it is retained, and what control users have over it, such as access, correction, and deletion rights.
For product teams, transparency should be treated as a core product design decision rather than just a documentation requirement. How clearly users understand what a system is doing directly shapes trust, adoption, and long-term safety in real-world use.
Building Consent, Compliance, and Fairness Into Systems
Because many ed tech tools serve minors, consent and compliance work best when they are built directly into product workflows rather than added later as separate requirements. Teams should think carefully about how consent is obtained and managed in practice. In institutional settings, for example, educators or caregivers often play a central role in account creation and authorization, creating more reliable consent processes than simple self-attestation models.
Laws and regulatory frameworks such as COPPA, FERPA, and GDPR provide baselines, but product teams still need to translate those requirements into everyday design decisions — including onboarding flows, data retention policies, and mechanisms that allow users to access, correct, or delete information.
As AI systems become more adaptive, fairness also becomes an important safeguarding consideration. Bias can emerge when certain learner groups are underrepresented in training data or when systems perform differently across populations over time. Addressing this requires ongoing evaluation, particularly as products scale and interact with more diverse learners and learning environments.
For product teams, this reinforces an important point: safeguarding is not a one-time compliance exercise. It is an ongoing process of evaluating how systems behave, who they work well for, and where additional protections or adjustments may be needed.
Conclusion: Designing for Trust
Many of the most important safeguarding decisions happen long before a product reaches learners. Decisions about what data is collected, where human oversight is maintained, how AI-generated outputs are communicated, and how systems are evaluated over time all shape whether a tool is ultimately worthy of trust.
As AI becomes more deeply integrated into learning environments, safeguarding cannot function as a one-time review or compliance exercise. It becomes an ongoing design responsibility — one that directly influences how safely, transparently, and effectively learners experience a product in practice.
