A range of tools to explore and assess the challenges and opportunities associated with emerging technologies in education. This is the first prototype for the tool which will be refined and tested over time with diverse stakeholders.
:Education Futures Studio]
EdTech Assessment Toolkit
Prototype 1 (October 2023)
Why is this toolkit needed?
EdTech is a rapidly growing industry across the globe. AI and proprietary edtech products and services are being rapidly integrated across schools, universities, and the sector (Gulson et al., 2022). Stakes become quite high when both immediate and longer-term inequalities (e.g. harms and risks) are embedded within these socio-technical systems. Shaping equitable AI and edtech to maximise educational benefit and minimise risks is urgently needed (Loble & Hawcroft, 2022).
Yet the inner workings of potential benefits and harms are mostly complex, hidden from view, and under-explored with diverse education stakeholders. A key problem is that edtech is often over-hyped and deployed with little scrutiny across a variety of essential educational services: school operations, teaching and learning, plus staffing. Addressing AI in education is vital to “realize key opportunities, prevent and mitigate emergent risks, and tackle unintended consequences” (US DoE, 2023, p. 3).
A range of frameworks, mechanisms, and initiatives identify the need for diversifying stakeholder expertise and public scrutiny of increasingly complex systems across society in collaborative ways (Swist et al., 2022). The development of education-specific toolboxes to explore the range of actors, scales, and expertise of edtech developments is vitally needed (Castañeda & Williamson., 2021).
Process and background
The toolkit prototype is an integration of insights from previous, and ongoing, Education Futures Studio (EFS) presentations, collaborations, dialogue, and workshops with cross-sectoral stakeholders, including: interdisciplinary researchers, educators, policymakers, students, teachers, education department representatives, campaigners, and advocates. Prototyping is a process for generating and testing ideas with material objects, which can inform new possibilities and solutions. While commonly associated with design and engineering fields, ‘prototyping’ is now being deployed more broadly as part of policymaking and transdisciplinary research projects focused on grappling with complex technologies and issues (Gils et al., 2021; Gengnagel Nagy & Stark 2016).
This prototype is informed by our interest in working with diverse stakeholders to better understand how emerging technologies can be assessed and audited. This EFS methodology is informed by a ‘technical democracy’ (Callon et al., 2009; Thompson et al., 2023) approach which focuses upon collective learning and experimentation about sociotechnical controversies. A policy, future-oriented and studio-style process is also key to this process which explores emerging public policy problems and incremental ways of diversifying participation and expertise across a range of organisations, locations, and publics (Kimbell, 2019).
How could this toolkit be used?
There is no formula, or prescribed order, to utilise this toolkit. Instead, we invite you to test it out and share with us how you engage with it in your particular context. This toolkit seeks to inspire an ‘on-yet-around’ (Fawns et al., 2023) focus about the use and effects of particular technologies, plus their interplay with dynamic, complex conditions. For example, the “looking forward’ and ‘looking backward’ examples indicate how particular stakeholder needs in the present interrelate with future needs and/or past concerns. In doing so, a multi-temporal and multi-scalar assessment of existing, and potential, uses of EdTech products and lifecycles in education can be facilitated.
Looking on-yet-around
Looking back
Exploring the complexity of emerging technologies and stakeholders across education and society.
Expanding stakeholder dialogue and decision-making between stakeholders about socio-technical issues and possibilities.
Engaging in collective learning, experimentation, and policymaking to make education-specific measures.
Looking forward
Looking forward
Alex is a primary school teacher who is often approached by EdTech vendors to trial a new technology for free. Alex would like to know more about how these technologies work and what risks they might pose before deciding whether to accept these opportunities.
JK is a student who is concerned about a biometric learning support tool which will be trialled soon with the senior school students. JK wants to find out if this has happened already in other schools and what students could do to have a say in whether or not the technology ends up being deployed longer-term.
Looking back
Greenlands High School has been using ‘x’ technology since 2015. In the past year, an increasing number of parents and students have raised concerns about the harms and risks associated with the system. The Parents and Citizens’ Association (P&C) wants to address these concerns, but doesn’t know where to start.
For the past three years, Principal TK has used BigTechCo’s Automated Essay Scoring software to grade all exams in his schools. The contract with BigTechCo is up for renewal, and TK is uncertain how to assess the benefits of the system vs the risks and harms.
Join our prototyping process
The EFS Technical Democracy Collective is interested to know how you utilise, share, or adapt the toolkit components for your particular issue, or organisation so we can update this toolkit prototype. To join our prototyping process, we invite you to:
Share your experiences of utilising this toolkit
What could be added, how it could be improved etc.
Your interest in future testing events, or collaborations, relating to the toolkit
Please contact teresa.swist@sydney.edu.au to share your experiences, ideas, and interests.
