Impact on pupil privacy

More specifically: if the processing of personal data is likely to result in a high risk to the rights and freedoms of the data subject, especially as a result of new technology. But what is the threshold for “high risk”? This is not always easy to know. The Data Protection Authority has prepared guidelines for the performance of these assessments, as well as a list of processing activities that always require a DPIA. There are several items on this list that are relevant for the AVT project:

• Data collected via third parties in conjunction with at least one other criterion.
• Processing of personal data using innovative technology in conjunction with at least one other criterion.
• Processing of personal data for evaluating learning, coping and well-being in schools or kindergartens. This includes all levels of education from primary school to secondary school and higher education. (Vulnerable data subjects and systematic monitoring.)
• Processing of personal data to systematically monitor proficiency, skills, scores, mental health and development. (Sensitive data or data of highly personal nature and systematic monitoring.)

In the AVT project, they had already performed a DPIA before the project was accepted into the sandbox. One of the sub-objectives for the sandbox project was to further refine this assessment, with suggestions from the Data Protection Authority on issues related to artificial intelligence.

As the sandbox project progressed, we acknowledged that there would not be enough time to carry out all of the activities on the project plan. We therefore had to deprioritise activities related to this sub-objective. As a result, we were only able to carry out one activity related to DPIA in the sandbox: a workshop with stakeholders, all of whom represented groups who would be affected by the learning analytics system in various ways.

We have also been made aware of potential data protection risks through discussions in other parts of the sandbox process. We briefly go over these findings, which are also relevant for a data protection impact assessment, in the section headlined “Other identified risks”.

Workshop with stakeholders

The workshop included pupils, parents, teachers and data protection officers from municipalities working with the AVT project. The event began with a joint presentation of the learning analytics system, followed by discussions in smaller groups – one with a combination of children and adults and one with adults only. Most of the risks mentioned had already been identified by the participants in the sandbox project, but discussions with stakeholders contributed to further clarifying these risks. Findings follow below.

Risk of altered behaviour/cool-down effect

If pupils change their behaviour when they are working with digital learning tools because they feel the learning analytics system is monitoring them, there may be a cool-down effect. The pupils were especially worried about the system monitoring how long it took them to complete the task. They pointed out that if the time they spent working on a problem was recorded, it could lead to pressure of solving the problems as quickly as possible, at the cost of quality and learning from problem solving.

Another example of a cool-down effect could be that the pupils do not feel as free to “try and fail” in their problem-solving, because everything they do in the digital learning tools is recorded and may potentially affect the profile built by the learning analytics tool.

If the introduction of an AI-based learning analytics system in education leads to a cool-down effect, the AI tool may be counterproductive. Instead of the learning analytics system contributing to providing each individual pupil with an education adapted to their needs, the individual pupil adapts their education and behaviour to the system.

Risk of incorrect personal data in the system

A fundamental principle in the General Data Protection Regulation is that the personal data being processed must be correct. Incorrect and imprecise data in a learning analytics tool could have direct implications for the profile of each individual pupil. This could, in turn, affect the teacher’s assessment of the pupil’s competence and the learning resources recommended for the pupil.

The learning analytics system collects data on the pupils’ activities from the digital learning aids used by the school. One potential source of incorrect data, which was discussed by the adult participants in the workshop, is when a pupil solves problems on someone else’s behalf. This has likely always been a risk in education, and there is no reason to believe that a transition to digital activities has changed anything in this regard. However, the impact on the individual pupil may be far greater now, if the data from this problem solving is included in an Ai-based profile of the pupil. As an example, the system may be tricked into believing that the pupil is performing at a higher level than they actually are, thus recommending problems the pupil does not yet have the skills to solve. This could be demotivating for the pupil, reinforcing the experience of not being able to master a subject or topic. A similar source of incorrect data is when a pupil deliberately gives the wrong answer, to manipulate the system into giving them easier or fewer activities. This, too, is a familiar strategy, used by children since long before the digitalisation of education. What both of these examples have in common is that the problems must be addressed both technologically and in raising awareness in general.

Risk of the technology causing the pupils unwanted stress

In using a learning analytics system, there is a risk that the distinction between a practice/learning setting and a test setting is blurred for the pupils. Teachers already use information from the pupils’ problem-solving and participation in class as a basis for assessment of what the pupils have learned. By using a learning analytics system, however, this assessment will be systematised and visualised in a different way, compared to the current situation. The pupils expressed concerns that there would be an expectation to show their “score” in the system to peers and parents, similar to how pupils currently feel pressure to share test results.

Other identified risks

Special categories of personal data

The AVT project does not plan to process special categories of personal data (sensitive data) in the learning analytics system. This type of processing requires a separate legal basis pursuant to Article 9. We discussed this issue in the sandbox, to promote dialogue and reflection around potential future needs, so that this could be addressed in the development phase to the extent possible.

Among other things, we discussed the possibility of the system being used to identify learning disabilities. Often, if the possibility to use a system for other useful purposes than what it originally was intended for exists, it is only a matter of time before it is.

It is also not unlikely that the education sector may be authorised to use artificial intelligence to uncover learning disabilities in the future. The earlier learning disabilities can be identified, the earlier the person with the learning disability can get the follow-up they need. The condition, of course, is that the municipality has a clear statutory authority and that the AI system has been developed responsibly, within the framework of relevant law.

The AVT project has been aware of this issue, and they plan to initiate measures to prevent the system from processing special categories of personal data. These measures include regular control of the algorithm and the data it generates. The solution does not currently have any mechanisms in place for blocking free-text responses from connected teaching aids, but the system does not process data from free-text fields in its learning analytics.

Processing of data about third parties

The processing of data about third parties was also discussed in the sandbox. This refers to the risk that municipalities process data about pupils’ parents, siblings and friends in the system. The municipalities have no statutory authority to process personal data about anyone other than the pupils.

The AVT project has described that they do not believe the solution could include information about parents, siblings or other third parties. Many of the measures implemented to prevent processing of special categories of personal data would also be relevant in this context. Such as not processing data from free-text fields in the solution. It would, however, be necessary to review this regularly. Further development of the system and integration with new service providers will entail a risk that third-party personal data may be processed through the solution at some point in the future. If so, it will be essential to identify if, and if so, when, this happens.

Decision support

The AVT project assumes that teachers will be using this system as decision support, and not as an automated decision-making system. This is relevant for the assessment of whether the use is subject to Article 22 of the GDPR. This provision grants the right “not to be subject to a decision based solely on automated processing”. There are several reasons why this was a topic of discussion in the sandbox. Among other things, users may want to use the system as a system for automated decision-making in the future. There had been several discussions within the AVT project previously, concerning the role of the teacher, and the importance of not marginalising the teacher with this learning analytics system.

Another risk is that the system is used as a de facto automated decision-making system, even if it is not designed to function that way. Some teachers may accept recommendations from the system without making their own assessments. This may be due to high workloads or a lack of knowledge about the algorithm, insight into how the system works, etc. We can also envision a situation where the recommendations from the system are so good that the teachers feel they cannot override the system.

In order to prevent a decision support system from slowly transitioning into a decision-making system, it is imperative to be aware of the risk and being conscious of this process. In addition, it will be important to implement measures to ensure that the system is actually used as decision support, such as the school owner developing procedures and training users on how to use the system, how it works and what the system’s recommendations are based on. This dissemination of knowledge is important for enabling teachers to consider the system’s recommendations in light of their own observations and knowledge of the pupils. Other relevant measures may include the implementation of procedures to reveal whether the decision support is real, or whether the system’s recommendations, in practice, are entirely decisive for the teacher’s decisions. To reduce this risk, the AVT project designed the learning analytics system so that the pupil or teacher has to actively choose whether to accept the system’s recommendations.

Similar issues were also discussed in the NAV project. The report from this project is available on the Data Protection Authority website.