Responsible partner for the piloting: University of Oulu
Contact persons: Hanni Muukkonen-van der Meer and Egle Gedrimiene
Piloting of learning analytics is a closely related to the piloting of other services and functionalities in CompLeap project. Thus, the timing, materials to be piloted, type of feedback and other details of the piloting depend hugely on the project goals, general piloting plan as well as development process of Compleap services, collaboration between Compleap project team and the developers and current legislation concerning personal information and data use in Finland and EU.
There are certain limitations to the piloting and testing of the recommendation system as well as competence visualizations imposed by the current Act on National Registers of Studies and Degrees in Finland (Laki valtakunnallisista opinto- ja tutkintorekistereistä, 884/2017). For this reason no real user data will be used in testing and piloting of the services. Mock up data from KOSKI services will be used in the piloting phase and end users will be able to try out the services without their own data but with the mock up data having close resemblance.
During the course of the project, research was conducted regarding possibilities to use various educational data sources and data types for the purpose of supporting learners’ educational decisions. Data model was created describing data sources, data types and pathways in Compleap services. The services are interconnected and are presented in the figure bellow followed by the more detailed description of each service.
Figure 1. Data model describing data flow
Data flow model in the Compleap project illustrates how data flows in the Compleap user services.
User comes to landing page and either logs in or uses system without logging in. If user logs in and has SSN (HETU) then all user related data is brought automatically from various data sources to dataset combination.
Dataset combination can be filtered by the user.
Filtered dataset is used as basis to semantic matching with KTO (KTO is abbreviation for Finnish words Koulutusinformaation, koulutustarjonnan ja Opintopolku.fi -uudistus).
After semantic matching phase user will get personal recommendations based on input data.
After dataset combination competence profile is showed to user (ESCO). (This is right from the "dataset combination".)
In case user do not login (s/he does not have Finnish SSN) then only one data set is automatically fetched for the user ("other competences") and this will be used for recommendations. This is a set of competences that user has selected from a list (national education classification).
Information about non-formal and informal education as well as interests comes from the manual user (learner) input. Specific data set is extracted and used as a data combination to visualize competences for the user and at the same time stimulate his reflection on his current competence situation. User can also manually select data combination and filters to get a content-based education recommendation. Best results are presented to the user where he can further select his favorites.
Information flow described above leads to two main services provided: Competence mapping and education recommendation.
- Competence mapping - there are certain competences acquired during the study process. These competences are usually described in the national curriculum and are important part of the modern world of work and study. Although they are learnt and specified in documentation many students don't have enough information about them or maybe haven't even heard about them at all. Compleap service provides an opportunity to see what competences in a form of study modules have been gained during the study process in educational institution. Mapping of Finnish national curriculum to ESCO competences is also done in this part of the project. However, possibilities to explicitly show these mapping results to users are still unclear.
- Education recommendation - education recommendations are calculated and based on similarity of content between user's profile data and education descriptions. Closest matches of between user's profile and education descriptions are presented to the user in ranking. Starting from the closest one to the less close and so on. User can mark some recommendations as favorite and this way generate more suitable suggestion.
The whole service is seen as a study guidance and support for decision making when choosing suitable vocational education and training. Information about own competences, previous studies and interests are gathered in one place and presented in a user-centered way to promote his reflection and guide him to studying possibilities and the world of education in general. Education recommendation is provided not as a solution but as an encouragement for the person to think and reflect on his interest and future educational and work-related goals.
At the moment, competencies visualizations and learner’s path visualization is left out of the further development scope of the Compleap project. Two main parts of learning analytics In Compleap are currently developed – competence mapping and education recommendation. More detailed information on these services, desk and user research is available is the following documents: competence visualization, education recommendation, summary of research activities in Compleap. Competence visualizations are currently left out of the scope of the project because of the lack of resources, and the gained competencies are provided in the form of the study modules. However, feedback about gathered visualization ideas will be collected form end users using low-fidelity prototyping. This will create a research base for further learning analytics development in the field of competence visualizations.
Timetable
As for now, piloting of competence mapping and education recommendation services is seen as a two-step process. End user piloting is very important for the development and proof of concept of the services. However, in some cases, this may not be feasible due to some limitations, e.g., maturity of technology, resources and priorities in development. Technical piloting will be important to test and improve technical details of services and piloting with end users will be necessary to see the final value, understanding and interpretations of the services for the end users.
2019, May, June, July
Technical piloting of Competence mapping -This part requires user’s previous education to be accessible from national database (KOSKI), integrated with national curriculum descriptions in e-peruste.fi and mapped to competence data from ESCO database (Classification of European Skills, Competences, Qualifications and Occupations). This would provide representation of Finnish vocational education and training curriculum in ESCO classification. However, possibilities to explicitly show these mapping results to users are still unclear and may only take the form of digital testing and not piloting with end users.
User piloting (students) - requires data flow in the Compeap system. It requires information to be available from specified databases as well as calculated algorithms to be functioning to provide individualized education recommendation based on user profile information. There is a need for this recommendation to be based on the content of the user profile and user behavior (favoring and marking some of the recommendations) and not on the choices made by other users. Interest ontology (from finto.fi) must be integrated and functioning as part of the recommendation system. Exact mathematical model of the recommendation system will be left for the developers to create. Available prototype will be tested with end users (people in need of guiding services), and feedback will be gathered about their experience and connection of these experiences to educational and learning phenomena.
As no user personal data will be available for the piloting phase, mock up/historical database will be used.
Expected outcomes - gathered data about end-user experiences (counsellees) and possibilities to use recommendation system for self-regulated educational guiding. Also information available about possibilities to use ESCO classification in learning analytics platform for guidance.
2019, August, September, October
Piloting of education recommendation
Technical piloting - The aim of this piloting phase is to test the recommendation system and its technical aspects, e.g. responsiveness to user requests, scalability, and peak load or reliability and others (Jannach, et al. 2011). This phase of the testing could be done without involving the end users and will be carried out as soon as development process of the recommendation system is advanced enough.
User piloting (counselors) - in this phase feedback will be gathered from the end end users (counselors) who used the application as a tool in a guiding session with their clients. Experiences of counselors will be gathered in a form of structured interview, collecting their needs and suggestions on further development of the guiding services and possibilities to scaffold it with educational technology.
Expected outcomes - data gathered to evaluate the value of recommendations to the counselors and further development possibilities for recommendation system.
Defining the user groups: the KOSKI database, only has data available from 2018 on the upper secondary and vocational education. This would limit our user group for the piloting considerably to the users who only graduated upper secondary school 2018 and those who started vocational education 2018. However, us no real user data will be used, these limitations are not critical. Nevertheless, the results of evaluating recommendation systems using historical data-sets cannot be compared directly to studies with real users and vice versa as data accuracy of real user preferences is not captured (Jannach, et al. 2011). Thus, it would be important to gather some user feedback and information needed for further development of recommendation system. To make user experience as close as possible to having own personal data in the services, mock-up data will be selected to correspond the most to the previously identified user groups (see the table of the user groups). Participants of the piloting will be selected accordingly to represent identified user groups. This will be done to ensure then the participants of the piloting are able to identify with presented data visualizations and education recommendations.
Methods
Two types of feedback could be gathered. One from the user and one from guidance counselors helping the student. Three types of interactions could be observed here:
- student - counselor
- student - content of education recommendation
- counselor - content of education recommendation
Survey and Interview question will be created using input from stakeholders seminars. These will be used to gather information on these interactions, providing qualitative and quantitative data.
Main questions when piloting the competence visualization / individualized education recommendations:
- How are the individualized education recommendations understood and interpreted by the users?
- How useful these recommendations are to the students?
- How useful these recommendations are to the counselors?
- What emotional, cognitive or behavioral response education recommendation elicit in the students?
Gathered feedback could also be related to the structural design and visual parts of the services as well as general users’ understanding, and interpretation of the suggestions given by the recommendation system