Abstract
The current study aims to assess the development of the knowledge generation program of the European Reference Network on Rare Endocrine Conditions (Endo-ERN) from its start in 2019 until December 2021, with special reference to webinars. We analyzed the number of webinars and live/postevent participants and whether participation and engagement of the attendees changed over time. A total of 30 (86%) self-prepared webinars comprising 300 h of knowledge and competence sharing were broadcasted (2019 – 3; 2020 – 13; 2021 – 14). A total of six webinars were broadcasted live prior to the coronavirus disease 2019 pandemic (https://endo-ern.eu/events/webinars/). The most active main thematic group (MTG) was MTG3 Genetic Disorders of Glucose and Insulin Homeostasis with eight (27%) webinars. Two (25%) MTGs fulfilled the goal to prepare at least two to three webinars per year. Patients were actively involved in 20% of the accounted webinars as both creators and presenters. The total number of live and postevent participants was 3023. The availability of the webinars after the live broadcast increased their outreach with a larger number of postevent viewers (n = 1629, 54%). Within the formal structured evaluation of the webinars, 40–85% of the participants replied on separate occasions and helped improve content. The free webinar access is among the perceived reasons for the rapidly increasing number of total hits to the Endo-ERN website. In conclusion, for its short existence, the Endo-ERN rapidly developed educational outreach, and further efforts to attract creators and learners are warranted.
Introduction
The European Reference Network on Rare Endocrine Conditions (Endo-ERN) was sufficiently installed in 2017 together with 23 other European reference networks (ERNs) for rare diseases (RDs). The organization and governance of Endo-ERN have been extensively explained and published (1). Endo-ERN is unique with its eight main thematic groups (MTGs) and four work packages (https://endo-ern.eu/specific-expertise/). Every ERN is engaged in the triple obligation of highly specialized health care, research, and education that may eventually decrease multiple unmet needs of RD patients (2). The great diversity of conditions, affected ages, and national and international support to the local RD health-care providers (HCPs) leaves those important tasks entirely to the own decisions of the ERNs.
The work package of Education and Training (WP1) of Endo-ERN had the major tasks to collect information, assess, align, and boost education and training, with a special emphasis on transition from the pediatric to the adult age care. It was created following the main principles of Endo-ERN that have proved to be sustainable so far – both pediatric and adult health professionals (HPs) and patient advocacy representatives' engagement (3). The baseline assessment of educational needs of HPs showed that 90% of the respondents supported Endo-ERN–coordinated activities and a common educational platform (4). Moreover, patients also showed preference to professionals’ derived information as well as social media/internet-based approach (5).
At the time of the creation of ERNs, the usual way of adopting new professional RD-related information was through self-learning, own HCP educational activities, and participation in physical meetings. The coronavirus disease 2019 (COVID-19) pandemic profoundly changed the learning reality (6). Various approaches have been implemented for early diagnosis of COVID-19 patients for clinical management and remote treatment but also for medical education (7, 8). Many providers started to offer technology-enhanced learning primarily as webinars thriving on representatives of specialties that could not properly work on a daily basis throughout the pandemic (e.g. pediatricians and surgeons) but mostly on the need to support professionals as much as possible throughout that health crisis. The training had to be remote and as inclusive as possible. Consequently, RD networks started to organize their first massive open online courses (MOOCs; https://www.futurelearn.com/courses/rare-genetic-disease). Endo-ERN recently engaged in such activities, and its first MOOC has enrolled 490 participants (https://www.futurelearn.com/courses/introduction-to-translational-research-for-rare-diseases). The accumulation of the so-called free open-access medical education resources is also progressing (9). All these evolving educational methods have large potential for ongoing peer review and integration of newly emerged knowledge but also require new types of quality assurance techniques for evaluation of the level of evidence and reliability that are continuously being developed (10). The boosting telemedicine tools and practice are also means of a potent knowledge generation (6).
In 2017, Heon-Clin concluded that RD networks would need a systematic knowledge generation plan (11). Since the beginning of Endo-ERN, it became clear that the diversity of the rare endocrine conditions is so large and also that the distribution among the affected children and adults is so variably expressed that more targeted efforts are needed for success. A major finding of previous studies concerning RDs is that the most critical single driver of looking for knowledge is a rare patient posing diagnostic and treatment questions (12). A collective endeavor to upgrade knowledge of a group of health workers supplying everyday care to rare patients and patients' personal interest in acquiring more sound medical information about their disease is a natural strive. Endo-ERN tried to support it through its virtual knowledge academy of webinars as a major part of the knowledge generation plan. The virtual academy started prior to the COVID-19 pandemic and continued throughout the pandemic without interruptions.
In the current study, we addressed the following two research questions: (i) How did the educational network develop from its start in 2019 until December 2021? and (ii) How did the attendees' participation and engagement change over time?
Methods
Webinar development protocol
From the beginning, the invitations to select topics and speakers were sent out to all eight MTGs of Endo-ERN. They were asked to prepare two to three webinars per calendar year, keeping close to the cited above principles of the network and to the findings of the baseline surveys about knowledge and educational gaps (4, 5). In more detail, this meant that the MTGs should try to create content of the webinars including relevant information about children and adults, transition from the pediatric to adult health care, patient views, and new diagnostic and treatment opportunities. Inclusion of patients’ stories, short videos, and unpublished material by personal decision of the presenters was also welcomed and actively sought. The final content was solely a decision of the MTG chairs and the presenters, based on the knowledge gaps, current interest (e.g. expected guidelines), and the MTG's wish. The choice of topics was not based on the actual prevalence of any disease.
Simultaneously, advertising of future Endo-ERN webinars started as a task primarily of the Endo-ERN project manager (PM). Initially, invitations were sent to the main representatives of the HCPs who were members of the network.
With time and with increasing interest, HCP representatives were asked to distribute invitations as much as possible in their own national networks of medical and affiliated HPs as well as relevant patient advocacy groups.
The developed modality of participation was as simple as possible – just a registration with the names, e-mail, and affiliation of the person, and was open to health-care professionals, patients, and students. The participation has always been free of charge regardless of whether the person’s HCP is a member of Endo-ERN. All registered participants received a registration link and two reminders – one on the day before the live event and the other 1 h prior to the event.
The webinars themselves were planned for a maximum of 1 h to 30–45 min for presentation and the rest of the time for questions, discussion, and closing remarks. The Endo-ERN PM acted as a host to every webinar in order to facilitate the process of answering questions and to tackle potential technical issues as well as to look after interactivity and participants' engagement.
The first webinar was broadcasted live prior to the COVID-19 pandemic, on 28 February 2019 to mark the Rare Disease Day, and a total of five webinars were held before the pandemic outbreak (https://endo-ern.eu/events/webinars/).
Webinar evaluation tools
With the improvement of distribution channels over time, the need to evaluate the webinars content appeared. This led to the change of the first chosen Internet instrument (Microsoft Teams) and then moving to the GoToWebinar tool, allowing for postevent statistical evaluation. We collected statistics from the web tool such as the total number of webinars and number of webinars per MTG, pre- and postbroadcast participation rate (number of participants), and the characteristics of the participants (HPs, patients, students).
All webinars were readily accessible also after the live broadcast through a very simple registration (name and e-mail address of attendee) as described above. A postwebinar evaluation survey (ES) was sent to all who accessed the webinars. The ES (Supplementary Figure 1, see supplementary materials section at the end of this article) was developed by the WP1 leads together with the PM and included 13 questions aimed primarily to improve future events and answer the rare endocrine disease community's expectations. Previous experience from face-to-face educational events organized by the group was taken into account.
Simple statistics were collected from the postwebinar ES – nationality, professional background, age group of the attendees, perceived webinar satisfaction as well as their preference for future webinar topics. We present findings from the tool statistics and ES, and a separate webinar facts and statistics as an example. The applied Internet tool gave the possibility to collect feedback only separately for every webinar. No summarized follow-up data could be analyzed so far.
We also present examples of the dynamics of the overall interest in the Network’s resources that are supposedly connected to the popularity of the Endo-ERN webinar program.
The current study does not use personal data and does not require ethical clearance by its nature. All data are anonymized.
Results
First and second steps
The first-year knowledge generation plan of Endo-ERN was ready by early 2019 and included endorsement of links to the e-learning resources of both learned societies: the European Society of Endocrinology’s e-learning platform and European Society for Paediatric Endocrinology’s e-learning portal (https://endo-ern.eu/activities/knowledge-generation/). The second step was the development of a program of regular webinars.
Number of webinars and their distribution across the main thematic groups
A total of 30 webinars were broadcasted between 28 February 2019 and 31 December 2021 (3 in 2019, 13 in 2020, and 14 in 2021). The representation of the organizing MTGs is depicted in Fig. 1. The most active MTG was MTG3 Genetic Disorders of Glucose and Insulin Homeostasis with 8 (27%) webinars, and the least active was MTG4 Genetic Endocrine Tumor Syndromes, with two (7%) organized webinars. Two (25%) MTGs fulfilled the predefined goal of preparing at least two to three webinars per year.
Distribution of webinars between the eight Endo-ERN main thematic groups (2019–2021).
Citation: Endocrine Connections 12, 9; 10.1530/EC-22-0512
Webinar participants
The webinars rapidly became increasingly popular. By the end of 2020, we had participants from many countries from all continents (Fig. 2).
Geographical distribution of the Endo-ERN webinar attendees (2019–2021).
Citation: Endocrine Connections 12, 9; 10.1530/EC-22-0512
The webinars were visited by physicians (adult and pediatric clinicians), patients and parents (according to the nature of the condition and the webinar content), students, and researchers. In 20% of the accounted webinars, patients were actively involved in preparing the content and participated during the broadcast.
For the analyzed period, the total number of live and postevent participants was 3023. From the beginning, there was a possibility to view the webinars after the live broadcast (Fig. 3). This increased their outreach with even a larger number of postevent viewers (n = 1629, 54%) than the number of the real-time attendees. The minimum number of attendees and viewers of a webinar was over 60 regardless of the fact that the presented condition was very rare.
Total attendees (live and postevent) per MTG webinar (by December 31, 2021).
Citation: Endocrine Connections 12, 9; 10.1530/EC-22-0512
Webinars with the highest participation rates
The seven webinars of MTG5 ‘Growth & Genetic Syndromes’ attracted a total of 1050 participants, and the eight webinars of MTG3 Genetic disorders of Glucose and Insulin Homeostasis attracted 571 participants overall. Among those, the most popular were the MTG5 webinars ‘Prader–Willi syndrome – PWS’ and ‘Diagnosis and management of Silver–Russell syndrome, a multidisciplinary care is necessary,’ and the MTG3 webinars ‘Wolfram Syndrome,’ and ‘Update on congenital hyperinsulinism.’
Webinar survey response rates and impact on future webinar planning
Since the start of the formal structured evaluation of the webinars (postbroadcast survey; Suppl. 1), between 40% and 85% of the participants replied on separate occasions. Apart from monitoring the level of interest and identifying problems, the very important selection of new topics from the suggestions of the viewers became possible due to the postevent survey. The ideas of involving more of the network patients led to inclusion of their stories and personal journeys within the condition health care in some of the latest webinars. Other ideas suggested by the attendees were involvement of basic science achievements, timely guidelines presentations, and further emphasis on transition of pediatric patients into adulthood care. For instance, the first MTG5 PWS webinar was followed by a second one, ‘Growth & Genetic Obesity Syndromes – PWS Clinical Management of transition of Care’ exactly 1 year later. The first MTG7 Sex Development and Maturation’ webinar on broader topics (https://endo-ern.eu/events/webinars/; Supplementary Fig. 1) was followed by a specific webinar on ‘Update on the Management of 46,XX Ovarian Dysgenesis and Primary Ovarian Insufficiency’ in 1 year.
As explained earlier, integrated statistics was not possible throughout the assessed period; therefore, we present the evaluation of one of the typical webinars – “Diagnosis and management of Silver–Russell syndrome, a multidisciplinary care is needed,’ broadcasted live on 24 November 2020 and available for additional viewers thereafter. These data were collected right after the broadcast. A total of 67 participants responded to the survey, predominantly pediatric specialists (58%), followed by patient representatives (16%), with only a few adult clinicians participating (6%). According to the answers, the webinar complied with one of the main aims of the Endo-ERN: 73% stated that they received information about the patient's transition (9% did not), and 92% stated they would recommend the session to a colleague. Over 50% of the attendees defined the information as ‘new.’ Furthermore, 87% of the respondents intended to join the next Endo-ERN webinar. Thus, it was not a surprise that by 24 April 2022 there were an additional 204 views.
Increasing popularity of Endo-ERN educational resources over time
Since the establishment of the webinars, the overall number of visitors to the Endo-ERN website and the total number of hits have increased (Fig. 4). After a considerable initial interest, a dip at the beginning of the COVID-19 pandemic was very obvious. With the pandemic mitigation already starting in 2021, there was again a substantial surge with a total of 474,659 website hits in 2021. It is interesting to note that from 1 January 2022 to 31 October 2022 the total number of website hits almost reached those for the whole previous year – 430,644. The linked media resources (Facebook, LinkedIn, and Twitter; Supplementary Fig. 2) are also becoming more popular among users, with Twitter demonstrating the largest number of followers (n = 1470; date of data 23 September 2022).
Total number of the Endo-ERN website (https://endo-ern.eu/) hits (2018–2021).
Citation: Endocrine Connections 12, 9; 10.1530/EC-22-0512
Discussion
Since the establishment of the first 24 European RD networks in 2017, the number of participating members (HCPs) grew from 900 to almost 1500 (https://ec.europa.eu/health/non-communicable-diseases/steering-group/rare-diseases_bg). Endo-ERN comprised initially of 71 members, while in 2022 the network expanded to overall 97 HCP members, 13 associated national centers, and 1 national coordination hub that now covers all 27 EU member states and Norway. Thus, increasing the efforts of the educational outreach was further needed, and at the same time, the general and in particular the teaching potential was augmented. The fast progress of the Endo-ERN webinar program exemplifies that this is possible.
The nature of the network in itself suggests competence, but expertise could not be taken for granted, and knowledge generation efforts are embedded in its very existence. The main challenge, as seen from the current study, is that the knowledge and training delivery depends on the engaged MTG chairs’ and affiliates' free will. With piling up of further tasks, sustainability is jeopardized (2), and thoughtful planning of efforts, both local and international, and structured support is needed. The uneven distribution of educational activities among the otherwise active MTGs was discussed (14), but no corrective measures have been undertaken so far. One possible way forward is the introduction of mandates for the MTG leaders, which is a pending discussion.
As Severin et al. (14) elucidate in their systematic review, education is not only raising awareness and providing knowledge and skills but also acting for societal support and fundraising through empowering individuals and groups. This is important especially for the middle- and low-income countries that just start to perceive the RD-related problems. It was postulated only in 2022 that learners from these countries would benefit more from easily available and free-of-charge digital education (15). In this aspect, the Endo-ERN Knowledge Academy approach for free webinar broadcasting has been inclusive and appropriate in an anticipatory mode from the beginning. Furthermore, easy access and worldwide participation is a step forward to decrease inequalities among RD patient care.
The possibility of live questions and discussions was actively used and obviously appreciated by the participants. The popularity of the webinars increased with the presentation of hot topics for the endocrine RD community, which fulfills one of the tasks of the networks – adequate education and training for all health professionals to make them aware of the available resources, thus overcoming border limitations and facilitating the mobility of expertise (http://www.rd-action.eu/eucerd/EUCERD_Recommendations/ern_recos.pdf 16).
Webinars and other online learning resources surmount the time constraints, demands of the working environment, and obligatory professional and personal/family tasks. They also decrease costs by saving on travel and accommodation. The Endo-ERNonline Knowledge Academy clearly demonstrated this with the achievement of a larger audience post live event. All this was noted prior to the COVID-19 pandemic that made the online training irreplaceable (for a review, see 8 and 16).
In the light of the current analysis, the sharp increase of both the number of webinars and participants underlined the increasing demand and skills of use of such resources. The published experience of the European Reference Network for Rare Neurological Diseases is very similar to ours (17). Its website’s visitors increased with time, especially during the COVID-19 pandemic, and with the use of other collaborative channels of advertising dissemination as well as social media. The Endo-ERN website’s visitors decreased sharply with the pandemic start most probably because of the changed curriculum of both health professionals and patients, but soon it increased again. A potential driver may have been the representation of pediatric endocrinologists who were not so overwhelmed with pandemic-related tasks.
The Endo-ERN Knowledge Academy with its developing webinar educational program is self-structured and evolving based on members' free will and perceived professional and patients' needs. While the current evaluation showed differences based on professional groups and conditions in terms of involvement and content creation, the overall participation was very satisfactory. The evaluation was useful for further improvement and upgrading the next MTG webinar. However, the usage of this tool is still not equally spread among all MTGs.
When asked in a former survey whether they would participate in creating new materials, younger patients/caregivers more often replied that they would engage in creating new informational materials and would also prefer to have more relevant information through the nowadays most popular media (5). We suggest that using such channels in the knowledge academy may help increase participation and patient engagement. Their further implementation is mirrored in the fact that a communication officer has been employed by Endo-ERN since October 2022.
The next webinars should broaden conversations with patients, interviews, videos, and personal experience sketches. The biggest challenge would be providing larger access for patients whose preference is to have materials in their own language (5), as well as assessing and addressing the patient-related outcome measures. Faster expertise creation would be sought; for example, a video of baby with PWS at the time of feeding, naturally including all necessary privacy precautions, would be more valuable than just its explanation (18). Further challenges are the need for frequent updating of the content, more focus on acquiring skills, improving competence, and refining professional performance, thus merging tacit knowledge with explicit validated knowledge (11), especially important for health-care specialists other than medical doctors. Trying to keep pace with progress, Endo-ERN should turn to more contemporary ways of education (https://mededu.jmir.org/themes/352-new-methods-and-approaches-in-medical-education) and try to encourage greater interactivity as a means of maintaining interest and increasing knowledge achievements.
A limitation of this study is the fact that we could not make a generalized statistical evaluation of the demographics of the participants, their specialty, if there were patients, their specific condition, and so on, which would have been interesting. This is even more important in view of the joint webinars with learned societies and other ERNs that started recently. Ways to overcome this shortcoming are currently being explored, with further modernization and refinement of the webinars' evaluation tools (19). Also, clues to improve the so-called soft skills as a basis of working in a multidisciplinary team should be a future part of the educational activities.
In conclusion, for its short existence during very challenging times, Endo-ERN has developed and achieved remarkable goals and activities in the field of knowledge generation. The existing and easily accessible free of charge content is readily available, and advertising the live broadcasting in a timely manner has been established. The expected common educational platform for all European RD networks would potentially facilitate the process in the near future.
Supplementary materials
This is linked to the online version of the paper at https://doi.org/10.1530/EC-22-0512.
Declaration of Interest
The authors report no conflict of interest in connection with this work.
Funding
This publication is supported by Endo-ERN. Endo-ERN is funded by the European Union within the framework of the EU4H Programme, grant agreement No. 101084921.
Acknowledgements
We would like to thank all MTG chairs, HCPs, and European patient advocacy groups who created content and organized the webinars, and continue to do so with technical support.
References
- 1↑
Pereira AM, & Hiort O. Introduction to Endo-ERN-scope and mission. Endocrine 2021 71 537–538. (https://doi.org/10.1007/s12020-020-02602-z)
- 2↑
Tumiene B, Graessner H, Mathijssen IM, Pereira AM, Schaefer F, Scarpa M, Blay JY, Dollfus H, & Hoogerbrugge N. European Reference Networks: challenges and opportunities. Journal of Community Genetics 2021 12 217–229. (https://doi.org/10.1007/s12687-021-00521-8)
- 3↑
Iotova V, Bertherat J, Mastorakos G, Hiort O, Pereira AM, & RARE ON. Endo-ERN on rare endocrine conditions: endo-ERN in its fifth year: a pinch of care, science, curiosity and new horizons. Endocrine Connections 2022 11 e220082. (https://doi.org/10.1530/EC-22-0082)
- 4↑
Iotova V, Schalin-Jäntti C, Bruegmann P, Broesamle M, Bratina N, Tillmann V, Hiort O, & Pereira AM. Educational and knowledge gaps within the European reference network on rare endocrine conditions. Endocrine Connections 2021 10 37–44. (https://doi.org/10.1530/EC-20-0480)
- 5↑
Iotova V, Schalin-Jäntti C, Bruegmann P, Broesamle M, De Graaf J, Bratina N, Tillmann V, Pereira AM, & Hiort O. Access to patient oriented information-a baseline Endo-ERN survey among patients with rare endocrine disorders. Endocrine 2021 71 542–548. (https://doi.org/10.1007/s12020-021-02654-9)
- 6↑
Hoofman J, & Secord E. The effect of COVID-19 on education. Pediatric Clinics of North America 2021 68 1071–1079. (https://doi.org/10.1016/j.pcl.2021.05.009)
- 7↑
Ozkara BB, Karabacak M, & Alpaydin DD. Student-run online journal club initiative during a time of crisis: survey study. JMIR Medical Education 2022 8 e33612. (https://doi.org/10.2196/33612)
- 8↑
Ahmady S, Kallestrup P, Sadoughi MM, Katibeh M, Kalantarion M, Amini M, & Khajeali N. Distance learning strategies in medical education during COVID-19: a systematic review. Journal of Education and Health Promotion 2021 10 421. (https://doi.org/10.4103/jehp.jehp_318_21)
- 9↑
Ting DK, Boreskie P, Luckett-Gatopoulos S, Gysel L, Lanktree MB, & Chan TM. Quality appraisal and assurance techniques for free open access medical education (FOAM) resources: a rapid review. Seminars in Nephrology 2020 40 309–319. (https://doi.org/10.1016/j.semnephrol.2020.04.011)
- 10↑
Brindley PG, Byker L, Carley S, & Thoma B. Assessing on-line medical education resources: a primer for acute care medical professionals and others. Journal of the Intensive Care Society 2022 23 340–344. (https://doi.org/10.1177/1751143721999949)
- 11↑
Heon-Klin V. European Reference Networks for rare diseases: what is the conceptual framework? Orphanet Journal of Rare Diseases 2017 12 137. (https://doi.org/10.1186/s13023-017-0676-3)
- 12↑
Vandeborne L, van Overbeeke E, Dooms M, De Beleyr B, & Huys I. Information needs of physicians regarding the diagnosis of rare diseases: a questionnaire-based study in Belgium. Orphanet Journal of Rare Diseases 2019 14 99. (https://doi.org/10.1186/s13023-019-1075-8)
- 13
White EK, Wagner IV, van Beuzekom C, Iotova V, Ahmed SF, Hiort O, & Pereira AM. A critical evaluation of the EU-virtual consultation platform (CPMS) within the European Reference Network on Rare Endocrine Conditions. Endocrine Connections 2022 11 e220281. (https://doi.org/10.1530/EC-22-0281)
- 14↑
Severin E, De Santis M, Ferrelli RM, & Taruscio D. Health systems sustainability in the framework of rare diseases actions. Actions on educational programmes and training for professionals and patients. Annali dell’Istituto Superiore di Sanita 2019 55 265–269.. (https://doi.org/10.4415/ANN_19_03_12)
- 15↑
Tudor Car L, Poon S, Kyaw BM, Cook DA, Ward V, Atun R, Majeed A, Johnston J, van der Kleij RMJJ, Molokhia M, et al.Digital education for health professionals: an evidence map, conceptual framework, and research agenda. Journal of Medical Internet Research 2022 24 e31977. (https://doi.org/10.2196/31977)
- 16↑
Sinclair PM, Kable A, Levett-Jones T, & Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: a systematic review. International Journal of Nursing Studies 2016 57 70–81. (https://doi.org/10.1016/j.ijnurstu.2016.01.011)
- 17↑
Brunelle Praschberger A, Post AEM, Hermanns S, & Graessner H. Establishing and boosting communication in the European Reference Network for Rare Neurological Diseases (ERN-RND): the impact of offering free educational webinars. Orphanet Journal of Rare Diseases 2022 17 89. (https://doi.org/10.1186/s13023-022-02209-9)
- 18↑
Xie H, Liu L, Wang J, Joon KE, Parasuram R, Gunasekaran J, & Poh CL. The effectiveness of using non-traditional teaching methods to prepare student health care professionals for the delivery of mental state examination: a systematic review. JBI Database of Systematic Reviews and Implementation Reports 2015 13 177–212. (https://doi.org/10.11124/jbisrir-2015-2263)
- 19↑
Nicoll P, MacRury S, van Woerden HC, & Smyth K. Evaluation of technology-enhanced learning programs for health care professionals: systematic review. Journal of Medical Internet Research 2018 20 e131. (https://doi.org/10.2196/jmir.9085)
