It is welcome news that individual appointments are being evaluated. Unfortunately we at UWS did not have solid evidence to convince management about the importance of individual consultations (other than the fact that other universities in Australia provide this opportunity for students) and they are now excluded from our workload. I look forward to the results of this endeavour,

thanks
Erst

Erst, this is something we've been meaning to do for some time! Each year we calculate the number of students who come to Academic Skills and Learning Centre for individual consultations, but the numbers, as has been mentioned in the literature, e.g., Peter Zeegers (2004), tell us little about the effectiveness of that form of teaching. How does the student view that form of teaching? What do they learn from an individual consultation that they can't learn elsewhere? What learning strategies do they find useful? Do they see a relationship between the individual consutlation and their learning? These are the types of questions we want to consider.

By the way, if you haven't read it, the recent article by Kate Chanock - "Valuing individual consultations as input into other modes of teaching" (2007) - is another valuable part of the argument that can be made for the usefulness of ICs in our teaching.

Cheers
Stephen

Hi Stephen,

Hope this reply isn’t too late to be of some help.

In late 2004, UQ asked students to evaluate its one-to-one sessions. On our questionnaire, which we asked students to fill in immediately after a session in the waiting room and drop in a box if they had the time to (about a 40% response rate), we had the following satisfaction questions with rating scales:

How well do you think your concerns were addressed in the session with the Learning Adviser?

Please rate how happy you were with the manner in which the session was conducted.

In both cases, if students weren’t reasonably happy, they were asked to explain why in open response questions.

We also asked them why they came, what they learned in the session, and perhaps very importantly in order to determine if one-to-one sessions have lasting value, asked those students who had been before what they had learned in previous sessions which they had been putting into practice. We didn’t, but I think it would be useful to ask a follow up question to the last one about what value to their studies the previous learnings had been (a couple of students provided that in response to one of the questions, saying they’d gone from getting passes to high distinctions!).

For this evaluation, I used open response questions because I wanted to see how the students themselves would describe what they were after and what they had learned, but I would not do so again for the following reasons. First, student answers were often brief and vague and so not as illuminating as one would desire. Second, they often used phrasing which an outsider might take the wrong way (e.g. saying they wanted their assignment proofread when in fact they wanted feedback on whether or not they had done what was expected and if not, where they were going wrong). Finally, in terms of what they learned in a session, they tended to focus only on their primary concern, ignoring other incidental learnings (e.g. almost every student I saw during the evaluation period was told about problems with their referencing, but only a couple of students identified referencing as being something they learned about in the session). Consequently, for all the questions mentioned above, in future I would develop a checklist of the typical sorts of things which might be addressed in a session and ask students to select as many of these things as were covered or they had wanted addressed (of course adding an open response “other, please explain” option).

Regards,

David

Your reply is not too late, David! We're hoping to get something in place for the second half of the semester, so we've got plenty of time to mull over what's been done elsewhere.

Asking students to give feedback immediately after a consultation is a big ask because, as you mentioned, they may not be in an ideal position to critically reflect on the usefulness of the consultation itself and what they gained from it. Students may not fully appreciate what they've learnt for some time after a consultation - some students may know right away, others may take a week/month to see the value of the consultation.

At ASLC I'm exploring how to go about sending out a survey at the end of the semester to students who have had a consultation within, say, the last two/three/four weeks. Privacy issues need to be dealt with and if we want to make public the results of the survey we will need to get ethics approval.

Developing a survey that combines both questions and checklists/scales (as you indicate) is the way to go I reckon.

What do others think?

Cheers
Stephen, feeling philosphical about the Crows and looking forward to 2008!

Hi Stephen,

From my experience there's no ideal time to survey. While some students don't have time immediately after an interview to fill in a survey, and there is as you say the issue of being able to digest what's happened, when we did survey as they walked out the door we got over a 40% response rate which is hugely better than any other response rate we've got with mail out type surveys. We tried to get around the problem of students not knowing as they leave what will prove of lasting value by asking students what they'd learned in previous sessions that they'd put into practice in between times (depends on having a reasonable number of repeat "customers" though).

So, I think it's a balancing act: high response but problems with quality, or very low reponse but perhaps better quality.

But I shall be very interested to hear how well whatever you decide to do ends up working!

Regards,

David

Hi all,
Well, during late November we sent out a survey to all the students who had an individual consultation at the Academic Skills and Learning Centre, ANU. Here's a little information on what we did, what we asked, and what we plan to do next.

We used an in-house electronic survey - APOLLO (ANU Polling Online). This is "a web-based application that allows users to create and conduct online polls (surveys, ballots, exams or electronic forms) ... without having to necessarily know anything about programming or databases. It provides easy to use tools for the analysis and reporting of poll responses and a comprehensive export facilities (in CSV, XML and SPSS formats)." All I had to do was skill-up to learn how to write and format the survey and then get ANU Legal to approve what we were doing.

For what we asked, see the attachment for the undergraduate survey. In addition to seeking student feedback on individual consultations, we thought the survey would be a great opportuntity to find out how students learnt about ASLC's existence and whether they used other ASLC services.

We created a grouplist of about 840 students, and once the exam period was over sent out a link to the survey. To encourage students to complete the survey we had 5 book vouchers ($50 each) to give away. The online survey was open for two weeks. At the end of two weeks, 149 students (85 undergraduate, 64 postgraduate) replied - not a bad result.

APOLLO sorts out the responses and allows responses to be filtered (e.g., if you want to look only at the responses from international students enrolled in the College of Asia and the Pacific, then it's possible to do that). Now we're in the process of looking at the student perception of the usefulness or otherwise of the individual consultation service.

After this, I think we'll do the survey again, but this time in a research context so that we can present our findings in a public way. For this survey, we chose to keep the findings in-house because we wanted to first test APOLLO and also test the appropriateness of the questions we had devised.

Expect some more updates on our progress during 2008.

Cheers
Stephen

stephen
thanks for the Dec posting in particular
I only just started looking thru this thread today, so after starting with your 1st posting , it was great to see the "what happened then..." part of the story
you've inspired me to have another go at doing some evaluation of our individ consults
like you, we have a long, proud tradition of such work here in Science @ La Trobe (since 1984 I believe), but what's important I think is to see what exactly it is that is being valued by the students. Sometimes a good evaluation can you show you things you weren't aware of, even when you think you know it all! - an occupational hazard of being in a job for a long time

I'm now thinking, as I write this, that maybe we (i.e LAS Science La Trobe) should gather a question or 2 from some of our colleagues in science who know us a bit, but who maybe don't know very much about the way we work in the individ sessions - might help us broaden our view

final point - our previous efforts to do surveys
1) originally we did the paper and pen thing - students collected a form (out of view of our office) straight after the session and then dropped it in a box down the corridor - from memory about 70% response (were we all less-surveyed-out then?)
2) hard copy mail-out to all students who had attended during that semester, sent out I think about a week before the end of classes - again, from memory, I think about 60 - 70% response. But that was also a few years ago
I look forward to hearing what happens next at your place

p.s. the emoticon I've used is labelled "shocked" - it is the closest visual image I could see to indicate "wide-eyed anticipation"