User Satisfaction of Children and Young People’s Service, Malta

Background

Professor Sir Ian Kennedy (2010) suggested that the only assessment of the quality of service in child health should be through service user satisfaction questionnaires.¹ To date only a few published studies have assessed the service users’ satisfaction of child and adolescent mental health services (CAMHS).² Promoting service user satisfaction may be an important aspect with the aim to reduce the dropout rates, which are reported to be between 30 to 60% of young people (YP) in CAMHS.^3-4 There is an association between YP and parental satisfaction and engagement and treatment completion.^5-6

Parent satisfaction feedback questionnaires have received criticism for the lack of psychometric evidence including validity and reliability.^7-8 They have also been criticised for not being comprehensive⁸ and this is related to content validity. One reliable and validated service user questionnaire is the Charleston psychiatric outpatient satisfaction scale, with a high internal reliability (alpha = 0.87).⁹

The aim of this study was to evaluate the current service users’ opinion of the national CYPS in Malta and disseminate them to commissioners and policy makers to be utilised for service development.

Method

Procedure

This study was a one stage survey to which all YP aged 18 years and under who were attending CYPS from the 1st to 30th November 2015 were invited to participate in. Data were collected over a one-month period.  This time period was decided upon based on the estimated projected sample size which was considered large enough according to published research¹⁰ for qualitative and quantitative data analysis. The national CYPS covered a population of 75,464 YP in 2015.¹¹

The inclusion criteria were all YP attending CYPS with a mental disorder according to ICD-10.¹² YP who had more than one appointment during the data collection period were eligible to participate on one of their visits only. All YP were offered the opportunity to participate or abstain, with the aim of reducing the chance of a selection bias.

Informed consent was obtained from every YP by the three nurses working at CYPS. The YP were then invited to fill in an anonymised questionnaire in either Maltese or English. The researchers (AS and NC) carried out the data collection and analysis had no direct contact with the participants, thus reducing the chance of an observer’s bias.

Questionnaire Development

The selected questionnaire was an adapted Charleston psychiatric outpatient satisfaction scale. It is comprised of a 5-point Likert-type response setup which optimises variability and predictive validity while decreasing positive response bias. All questions followed the hospital governance protocol and contained no unethical questions or questions which may negatively affect the mental state of the YP. The English version had a readability score of ages 13-14 years and the questionnaire was translated by a professional translator into Maltese. The questionnaire involved 13 questions with a Likert scale and three open ended questions at the end of the questionnaire.

A pilot study was carried out on 10 YP, to assess the acceptability, readability and reliability of the tool. The YP and carers were then asked to provide written feedback on the three mentioned points above. Both the English and the Maltese version of the questionnaire were found to be acceptable with good readability by all YP and care givers.

Analysis

A mixed method analysis was carried out using both qualitative and quantitative data analysis. The data from the questionnaires were inputted into a password protected Excel 2010 spreadsheet.

All quantitative data were grouped, and correlations were carried out using Statistical Package for the Social Science.¹³ The three variants with the highest correlation coefficient to overall quality of care were entered into a logistic regression to test whether the three independent variables were significant in rating the overall quality of care received.

The qualitative data were collected from the three open ended questions which inquired; i. how the service could be improved? ii. what would YP and care givers make sure such a service would include and iii. any other comments that they had. A thematic analysis as written by Braun and Clarke¹⁴ was used to analyse YP’s opinions as a combined group. In this thematic analysis four major themes were identified: clinical, administrative, environment and location comments.

Results

The target population was 216, five people refused to participate, either due to literacy difficulties or needing to attend to their child’s needs as a priority, thus the sample population was 211 (97.7% response rate). The average age of attendance was 9.97 years (SD 3.34, 95% CI ± 0.45) and all the YP were accompanied by a parent or care giver. 112 (53.3%) chose to answer the questionnaire in Maltese.

Figure 1

Results for the qualitative part of the questionnaire are presented in Table 1 and Figure 1. The overall quality of care was significantly correlated with respect shown for YPs' opinions about treatment (r=0.539, p<0.001), matching of the treatment plan to the YPs' individual needs (r=0.320, p<0.001), and the helpfulness of the services YP received (r=0.618, p<0.001).

Multiple regression analyses were used to test if respect shown for YPs' opinions about treatment, matching of the treatment plan to the YPs' individual needs and the helpfulness of the services YP received significantly predicted overall quality of care. The result indicated that the three predictors significantly account for 48.6% of the variance (R2=0.486, F(3,184)=57.93, p<0.001). Matching of the treatment plan to the YP’s individual needs and the perceived helpfulness YP received from the professionals within CYPS significantly influenced the perceived overall quality of care (β 0.325, β 0.361 respectively), p <0.001, that the YP received. However, respect shown for YPs' opinions about treatment was not significantly predictive (beta 0.114) p=0.135.

Thematic Analysis

Service users felt that more guidance, such as parental skills and children’s groups were needed to help them understand and deal with their children’s behaviours, Figure 2. It was emphasised that better collaborative communication including education was required and as a result, professionals can then prescribe accurate individualised treatment to meet their needs. Having access to a crisis team and helpline were amongst the suggested needs for a more holistic and efficient service.

Figure 2

36 (12.6%) YP said that review appointments were too infrequent and furthermore, having reviews carried out by a different professional on each subsequent review reduced continuity of care and hence the chances of building a meaningful therapeutic relationship with any professional, Figure 3. 20 (7.0%) YP said that the waiting lists for psychologists and doctors were perceived as being too long, some recommended an increase in the number of trained staff to patient ratio. 15 (5.2%) YP stressed the importance of professionals adhering to their respective appointment times. Lastly, it was suggested that clinic hours should be extended so that YP would not have to miss school to attend appointments and care givers not to need to take time off work.

Figure 3

The appearance and facilities in the waiting room were heavily criticised (n=50, 17.5%), YP asked for new and appropriate books to be made available in the waiting room, for there to be age appropriate toys, a larger TV screen, computers, Wi-Fi available, a food and drink machine and an overall warmer and age appropriate environment. 31 respondents (10.8%) reported to finding parking facilities difficult to access.

Conclusions

To our knowledge this was the first service user satisfaction evaluation carried out for the national CYPS, Malta. The overall feedback from the YP and their care givers was positive with regards to the overall quality of the service that they received. 50 (17.5%) YP commented positively and expressed their appreciation towards the service offered at CYPS. The main finding which support this included; YP felt that CYPS provided an individualised treatment plan which was tailored to their needs and that the staff worked collaboratively with the YP to achieve their treatment goals.

However, a substantial number (n=36, 12.6%) of YP and their care givers commented that waiting times for appointments were too long. Another recent service evaluation from the same CYPS reported the mean waiting time for the first specialist review was 301 days (CI ± 34.4, range 0-800) in 2014.¹⁵ Therefore, CYPS waiting times did not follow guidelines as recommended by the Local Delivery Plan Standards target in NHS Scotland, which reports that a child should not wait for more than 18 weeks¹⁶ from referral to treatment. This could be achieved by increasing the trained staff to patient ratio and investing in treatment pathways with target waiting times and auditing the service regularly.

The need for better communication between services was emphasized by the responders. To address this limitation CYPS staff have started carrying out more school visits, thereby liaising with the educational system, thus assessing YP in their natural environment to provide a better understanding of the YP’s needs resulting in a better outcome of care provided.¹⁷ Furthermore, there has been more emphasis on interdisciplinary communication between paediatricians and clinicians working at CYPS. This has been facilitated by having trained paediatricians working at CYPS and having an exchange training programmes between psychiatry and paediatrics of placements for trainees.

One important theme from this thematic analysis was the absence of continuity of care and therefore the difficultly to establish a therapeutic relationship between the clinician and the YP. This may be the result of the system set up, where by a YP is reviewed by a different doctor at each subsequent visit to CYPS. This limitation in service provision could be managed by each clinician having their own caseload and the YP is followed through from beginning to end, ensuring continuity of care. The study showed that if the YP’s individual needs and treatment plans are matched, then YP would feel that the service was helpful, and this significantly predicted overall quality of care.

Although the opening office hours and the location of CYPS were rated as overall good, the authors wonder if a barrier to service use is actually the opening times. Appointments are all given during school hours, this means that YP would need to choose between school and the appointment at CYPS.¹⁸ Furthermore, only referrals made by doctors are accepted to CYPS, meaning other clinical professionals working in mental health, education and care givers themselves cannot directly refer to CYPS. Published research reports that less than half of YP with a mental disorder actually access a mental health service,¹ facilitating the referral process procedure may therefore increase access rates. It was hypothesised from the complaints made by YP about the location of CYPS that there is a possibility that YP would prefer smaller community CAMHS provided in their local area rather than just having one centralised service for the whole island of Malta. However, this would involve a strategic service development plan with costs to cover a substantial increase in staff employment and training.¹⁹

Parking was rated poorly, and this may be another barrier to attending CYPS. Poor parking facilities may be one of the reasons for parents to choose not to attend CYPS and opt for other independent options available on the island.¹⁸ However, from the logistic regression, no significant correlation between parking and overall quality of care was found. Despite parking being rated most poorly in the quantitative and qualitative responses, this did not significantly contribute to the overall quality of care.          

From the information gathered in this study and extrapolated by the authors, it is hypothesised that a substantial number (data still not available in Malta) of YP choose to access independent CAMHS rather than the NHS CYPS. Reasons for the above could possibly be the result of the poor environment and location of CYPS together with unacceptably long waiting lists and the lack of possibility to engage with the same professional. Further contributing to the above hypothesis may be the lack of stigma, shorter waiting lists, age appropriate environment and the ability to self-refer associated with accessing private services.¹⁹ This hypothesis could be tested in future studies on this service.
         Another theme mentioned was the absence of age appropriateness of the environment of CYPS (n=50, 17.5%). Moderate associations were found between the subjective experience of the YP attending CAMHS and their evaluation of the physical environment. Relatively small changes, such as adding a piece of artwork can help to improve the general perception of CYPS. The relationship between the physical environment and patient satisfaction suggests a potential for policy makers and commissioners to improve the overall service provided.²⁰ Whilst appreciating the difficulties in providing an appropriate environment to meet the wide age range of YP reviewed at CYPS, in the published and unpublished literature, YP have pointed out the need for a “happy looking”, well-lit, comfortable²¹ and age appropriate environment.^22-23 Lastly an innovative concept which may be adopted to help YP attend CYPS is the introduction of static exercise bicycles, which could be placed in the waiting rooms and could be used to charge the electronic device of the YP or their carer. This could keep YP occupied while waiting, whilst help to reduce the ever-increasing problem of obesity in YP in Malta.^24-25

Strengths and weaknesses

This study was a one stage survey with a large sample population and with a high response rate, which was not limited by a selection or observation bias and data analysed by using mixed method of analyses. However, no demographic data (except for age) was collected, the hypothesis that asking for limited personal identifiable information would increase the acceptability by the YP to participate and reduce the chance of a response bias by a Hawthorne effect. This study was limited by a response bias given that YP who do not attend CYPS were not included in this service evaluation. The authors (AS, NC) were blinded to the responders and recommend that as the next step from this service evaluation, interviews with the YP would be carried out, through independent research assistants, thus increasing the depth and quality of information gathered.

Acknowledgements

Mr Victor Bonanno, Ms Claire Marie Cachia Attard, Ms Anthea Ciantar, Ms Alexandra Falzon, Ms Gertrude Frendo Cumbo, Dr Kristina Vella

References

1. Brown A, Ford T, Deighton J, Wolpert M. Satisfaction in child and adolescent mental health services: Translating users’ feedback into measurement. . 2014;41(4):434-46.
2. Garratt AM, Bjertnaes OA, Holmboe O, Hanssen-Bauer K. Parent experiences questionnaire for outpatient child and adolescent mental health services (PEQ-CAMHS Outpatients): reliability and validity following a national survey. 2011;5(1):18.
3. Kazdin AE, Mazurick JL. Dropping out of child psychotherapy: Distinguishing early and late dropouts over the course of treatment.1994;62(5):1069.
4. McNaughton D. Measuring parent satisfaction with early childhood intervention programs.1994;14(1):26-48.
5. Shapiro JP, Welker CJ, Jacobson BJ. The youth client satisfaction questionnaire: Development, construct validation, and factor structure.1997;26(1):87-98.
6. Stallard P. Parental satisfaction with intervention: Differences between respondents and non‐respondents to a postal questionnaire.1995;34(3):397-405.
7. Gerkensmeyer JE, Austin JK. Development and testing of a scale measuring parent satisfaction with staff interactions. . 2005;32(1):61-73.
8. Ayton AK, Mooney MP, Sillifant K, Powls J, Rasool H. The development of the child and adolescent versions of the Verona Service Satisfaction Scale (CAMHSSS). . 2007;42(11):892-901.
9. Pellegrin KL, Stuart GW, Maree B, Frueh BC, Ballenger JC. A brief scale for assessing patients' satisfaction with care in outpatient psychiatric services. . 2001;52(6):816-9.
10. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. 2004;10(2):307-12.
11. Eurostat. Statistical themes - Statistics Explained. 2016;2016.
12. Organization WH. The ICD-10 classification of mental and behavioural disorders: diagnostic criteria for research. World Health Organization; 1993.
13. Hilbe JM. Logistic regression. In: International Encyclopedia of Statistical Science. Springer; 2011. p. 755-8.
14. Stark D, Thomas S, Dawson D, Talbot E, Bennett E, Starza-Smith A. Paediatric neuropsychological assessment: an analysis of parents’ perspectives. . 2014;5(1):41-50.
15. Saliba A, Agius D, Sciberras E, Camilleri N. A population service evaluation of the Attention Deficit Hyperactivity Disorder (ADHD) pathway of Children and Young People’s Services, Malta. 2018.
16. Hookway G, Lead SQI. HIGHLAND NHS BOARD.
17. Pettitt B. Effective joint working between child and adolescent mental health services (CAMHS) and schools. Citeseer; 2003.
18. Koerting J, Smith E, Knowles M, Latter S, Elsey H, McCann D, et al. Barriers to, and facilitators of, parenting programmes for childhood behaviour problems: a qualitative synthesis of studies of parents’ and professionals’ perceptions. Eur Child Adolesc Psychiatry. 2013;22(11):653-70.
19. Bell CC. Commentary on the Child-Adolescent Mental Health Services (CAMHS) Conference—Nashville, TN, September 2009. 2010;37(1-2):135-9.
20. Rogers S, Edwards SJ, Hudman P, Perera R. The importance of the physical environment for child and adolescent mental health services. 2016;11(1):33.
21. Mitchell-Lowe M, Eggleston M. Children as consumer participants of child and adolescent mental health services.2009;17(4):287-90.
22. programme,NHS England/Medical Directorate/Parity of Esteem. Model Specification for Child and Adolescent Mental Health Services: Targeted and Specialist levels (Tiers 2/3). 2014.
23. Camilleri N, McArdle P, Newbury-Birch D, Stocken D, LeCouteur A. A case control and follow-up study of “hard to reach” young people who also suffered from multiple complex mental disorders. 2017;41:S82.
24. Grech V, Aquilina S, Camilleri E, Spiteri K, Busuttil ML, Sant'Angelo VF, et al. The Malta Childhood National Body Mass Index Study: A Population Study. J Pediatr Gastroenterol Nutr. 2017 Sep;65(3):327-31.
25. Pandian R. Human power conversion system based on video game play with exercise bicycle. 2005.

Figure

Test image