Course Write-up: Gathering feedback

Hello! It's been a while since I've written up any courses I've been on, in part because I am very lucky that I work in a place where we share what we learned on courses at our weekly staff meetings so I haven't felt the need to try to reflect or assimilate in writing. But it's always beneficial to write these things down for my own sake as well as sharing them more broadly, so I thought I'd try to make more of a habit of it, at least when I feel it would be useful to do so for my own practice. I'm working on a some other work-based and professional development habits, which I hope to talk a bit more about in an upcoming post. Meanwhile, here's a write up from a course I went on this week.

Sally Stafford's recent session on gathering evaluation and feedback was one I've been looking forward to for a while. As someone who teaches a fair amount, I often rely on the course leaders to pass on feedback from their students about what I might improve, and we are still working on developing strategies for consistently getting evaluations from the training sessions we give. While I really enjoyed this course, it was a bit of a stretch to think of ways of incorporating it into my own practice as there was a heavy bent toward feedback on exhibitions and outreach events rather than training sessions. Even so, it certainly got me thinking about creative ways of framing questions and assessing learning outcomes.

The first point Sally made was that when people think about feedback, they're often only thinking about gathering opinions after the fact. However, effective projects look for feedback throughout, from the initial development phase. By doing this you can ensure that you are delivering content that people really want, in a way they want. Teaching sessions have this built in, in that the process is inherently iterative: ideally, your training gets better each time you do it based on the feedback you've had before, and there is no "final" product to get feedback on. However, it is useful to think about impact in the same way museums do. What have people taken away from my session? What was the impact on their practice? Impact is a fiendishly difficult thing to measure, but various sectors are under increasing pressure to demonstrate measurable impact to justify funding, staff and other resources. This is not to say that everything needs to be reduced to a number, say "72% of participants said this training was Very Useful". I'm always more inspired by individual comments, like, "I learned a lot, thank you!" or "I never knew librarians knew so much about this stuff!" But in large volumes that becomes more and more difficult to parse and if you are involved in a project that requires you to justify funding, you may be dealing with people who find the numbers much more compelling than a few glowing remarks when presented in a report.

GLOs and designing questions

Central to Sally's process were the Generic Learning Outcomes, a framework used by the MLA to assess learning in non-classroom contexts. While much of the discussion around the GLOs focused on the context of exhibitions or outreach to school groups, I found the framework to be a useful prompt to think about what questions one could ask to gather feedback about different facets of learning.

• Knowledge and understanding: While this is fairly straightforward, asking bluntly "How much/what did you learn?" is not necessarily going to give you brilliant feedback. Any parent who has asked their kid what they learned in school today will be familiar with the non-commital shrug followed by that slippery syllable, "Stuff". The group talked about potentially asking for one thing that stuck out, one fact for example. Since I often do follow-up sessions or series of classes, I could always ask at the beginning of a class for one thing they learned in the previous session for example.
• Skills: This is a tricky one to get verbal feedback on, but could be tested through doing an activity before and then after the session. An approach that Sally used for our session was a target, where we rated our confidence with evaluation before and after the session using sticky dots placed correspondingly on the target. I think confidence is a good operative word when asking people to self-report on their skills. 
• Attitudes and values: Another tricky one to ask about, as Lucy was tactfully explaining, as it has to do with subjects that people are sensitive about. Sally offered the example of, "Are you a bigot?" as a potentially insensitive way of gathering feedback about this aspect. :) My content is often very value-laden. I talk pretty openly about my mistrust of metrics as a good indicator of the quality of an article, about the flaws in the peer review process and advocate passionately for Open Access publishing. Rather than gathering this in the form of feedback after the fact, I usually seek to have a discussion during the session where people are invited to share their points of view.
• Enjoyment, inspiration and creativity: While I don't think I'll collect finger paintings from my students, it's worth thinking of creative ways people could respond, especially if they have felt inspired by something in a session. I refer to this facet later as the 'Ah-hah' moment and discuss it in more detail.
• Activity, behaviour and progression: Once again I do often have the opportunity to find out what people learned from a previous class and I think I could take greater advantage of that to see if behaviour has changed in response to something I've taught. I usually ask if anyone has been using a particular tool or technique and ask for feedback but perhaps I could do this in a more structured way.

Creative feedback methods

I'm obviously still mulling over how to ask for feedback and how to record it, but the session was certainly not short of ideas. Many of them would suit a UX context better as they're fairly involved, e.g. focus groups, behavioural mapping, observation, interviews etc. Some suit different audiences better than others, for instance role-playing or drawing would be great for kids but I somehow doubt I could get a room full of stressed MPhils to see the value of such tasks (as much as I believe that creative endeavours are good for stress levels). There are some ideas I'm tempted to use, however. For shorter sessions on a particular tool or skill, I'd love to adapt the target method to show the change in confidence levels. For series of sessions I'd like to build feedback into subsequent sessions and help people reflect on how their practice of academic research is developing. It's definitely gotten some gears turning in my brain regarding how I could gather feedback beyond my usual post-its at the end asking for one thing they've learned and one thing they'd improve.

Ethics and accessibility

I wanted to raise a couple of issues that didn't come up during the course. First, gathering data from people I think it's essential to speak to someone in your institution that knows about research ethics. They may say that no further approval is needed, but the moment you start gathering artifacts or quotes from people, start observing their behaviour or start intruding on their time it's important to think about the ethical considerations.

• Is participation voluntary?
• Have you informed people that they're being watched?
• Have you informed them of how you will use their data and do you have a plan to follow through with that?
• Who will have access to the feedback they give you?

Unless told otherwise, people have a reasonable expectation of anonymity when taking part in studies. It doesn't have to be a signed consent form in every instance and can be very light-touch. For example, I plan on adding a quick verbal disclaimer when I'm asking for feedback that it's anonymous and any feedback they give us will be used to make our training programs better. There is a blurry line between user experience research and feedback and I would think that it would be good to err on the side of caution and consult with someone who can give guidance on what you need to tell participants and how to keep the data in a safe and anonymous way.

Similarly, it's important to think about comfort levels. One method discussed for use with teaching sessions was task-based feedback, for example acting exercises to gather feedback during a session. It was mentioned that adults were likely to be self-conscious about this, but that it would be engaging for children. I agree but I think it's important to be aware that if this is built into the curriculum of a particular course or training session it may not be accessible for people on the autistic spectrum or other social differences. By way of making course content equally accessible to all, I would be interested in finding a way for students to opt in rather than making it a requirement, or seek other ways in which you could gather similar feedback.

Accessibility sprung to mind again when we looked at feedback methods using red, green and amber coloured pieces of card to let the instructor know how confident or engaged participants felt. Again, alternatives that are accessible to colourblind participants would be useful to prepare ahead of time. These are just a few examples - there are many ways to build accessibility into your feedback process if you take the time to consider who is being excluded by the method you have chosen.

Ah-hah moment

I think my favourite concept from the course (and my own 'Ah-hah' moment) was the idea of focusing feedback on what inspired someone, one idea they'll take away from the session or one lightbulb that lit up during the session. Even if a student in one of my sessions paid me pretty much no attention but they had an 'Ah-hah' moment regarding their own work while sitting there, I feel like at least I provided them the space to get that inspiration and I'd love to know about it.

For kids visiting an exhibition, they may not grasp your thesis but they will certainly remember the taxidermy pigeons because they're surprising. Or they might remember that, like Charles Darwin, they always keep a journal too and like drawing the animals they see. Similarly, my students may not remember everything I tell them about Data Management, but I hope they remember the story about the guy who lost 6 months of work when his laptop containing his PhD and the backup disk, both in his rucksack, were stolen in a pub. Or when looking at conference posters for design tips, maybe a student will finally figure out what methodology she'll use for her own dissertation (this literally happened in my class yesterday). I don't really mind if she took on board less of what I taught as long as she left my class excited about something to do with her work.

I think beyond tips for improving the actual content or timing of the sessions this will be the focus of generating feedback. It provides students the chance to reflect on what they're excited about, which will reinforce whatever inspiration they've had, and it gives me a window into what material students are resonating with. On a personal development level, I've just started a weekly reflection/accountability thing where I write about what went well vs. not so well, which includes noting down my own 'Ah-hah' moments. It's really helpful to try to capture what's inspiring you from the courses you're taking or the books you're reading as it makes it easier to remember and therefore implement any changes that you might think of as a result (especially if, like me, you're consuming so many great articles and podcasts that often times great ideas are driven out and forgotten, no matter how inspiring they were).

So, to try to synthesise a write up in which I concluded pretty much nothing, it was a good session and it's definitely got me thinking. These concepts might go on the back-burner for a bit but I can have a look at this post later on when I'm developing courses to see if it prompts any more 'Ah-hah' moments. 

Library Inductions: Who, What, Where, When, How?

Do you know what an unconference is? Because I didn't until I went to LibCAMp, a Cambridge-based unconference on 21 July. So don't feel bad. (I still have to remind myself about the lower-case 'p'.)

I'm sure others will blog about the event, the various participant-led discussions, and of course there are always various Storify summaries to get a sense of the big picture. I specifically wanted to write my thoughts on the session I attended on library inductions, though, not because it was the most important talk of the day (the discussion led by two members of staff from the SPS library of what we as Cambridge Librarians can do to ensure services and resources are available to disabled students was - I hope - a major turning point on this important issue), but because inductions are part of what I do and for a couple years now I've been wondering how I can make them better. For this reason I thought it would be good to try to reflect and collect my thoughts on the session and maybe generate some ideas that would be doable.

Invaluable to get student engagement/co-operation with your inductions - how best to manage this? #camlibcamp

— Diana Sprints (@Selenographics) July 21, 2015

Library induction tours are the first chance we get to talk to students about the library. We generally do four tours in the first week of term that last half an hour each, and attendance is entirely optional. Usually we'll get one group of around 25-30 and the rest have around 3-8 students. I've been following the format I learned from longer-serving members of staff, whereby we show them around, show the catalogue, explain borrowing, fines, rules for use and so on. The obvious problems with this are a) it's not very engaging - we try to ask questions, tailor the tour a little bit to the subjects people are reading etc. but not very much - and b) the first week of term for first years is a time of information overload, so I'm not sure how much information is actually sinking in from those sessions. Furthermore, the uptake is not great so come Lent Term there are a lot of people who still don't really know some basic things about how to use the library.

The pitch for the LibCAMp session was to share ideas on how to make inductions more engaging, so I was very keen to attend and hear what people might suggest.

Deep in discussion about library inductions at #camlibcamp pic.twitter.com/TAupPoIwxn

— Claire Sewell (@ces43) July 21, 2015

Innovative forms of induction other libraries use include scavenger hunts, peer-led inductions, video/Vine inductions and hands-on sessions. Games were also mentioned as a possibility and in addition to SPRI's scavenger hunt, murder mystery style games were mentioned as something that other libraries around the world are doing. One of my favourite tips was to tell students that everything on the induction tour was something that a professor hadn't known, thus making it clear that it's normal not to have everything figured out and hopefully reducing the fear of asking "dumb" questions.

I was very pleased when the discussion moved toward User Experience (UX) and ongoing support. Many people agreed that inductions should not be thought of as a one-off session, but that we continue to communicate induction information through leaflets, signage, one-to-one troubleshooting sessions and so on.

Library inductions are not seen as most important to students. They shouldn't be one session, it's a continuing learning process #camlibcamp

— Rachel Claire Walker (@phys_librarian) July 21, 2015

In many ways the librarians present felt a tension between wanting the library to speak for itself - so that users could come in and accomplish their goals without ever needing intervention from library staff - and the desire to tell users what we think they should know. The problem is that we don't know what the gaps in knowledge are. Has everyone on the tour used a catalogue before? Have they already been told about search strategies? It's hard to know who will be bored by this basic information and who will be hearing it for the first time.

But the information overload of the first week – a factor over which we have very little control – may be a more critical factor in how students respond and how much they remember from the induction. That’s why it’s important to have induction information available at the point of need as well, or at least in a format that students can come back to when needed (like a postcard in their Freshers packs or a series of YouTube videos). The library should ideally be easy to navigate and the design should be unobtrusively helpful at the right points, meaning that inductions are not necessary in order to interpret the service.

I have been thinking about other kinds of induction that we might do and when we might do them. By the beginning of Lent Term, for instance, students will have gotten over the initial information overload and will have had personal experiences with library services. This seems like it would be a good opportunity to add some tips and tricks that will help students navigate the information landscape at Cambridge. I've been developing just such a course, adapted from Emma Coonan's "Here Be Dragons" presentation into a (hopefully) engaging look at "How to make the most of Cambridge Libraries". I would ask participants what barriers they've encountered using library services and tailor the session to address those issues, or if they don't highlight any issues then I can simply talk about tips and tricks to make better use of the library system at Cambridge. The problem with this approach is that one confusing, frustrating or otherwise negative experience in a library during Michaelmas Term may be enough to turn a library user into a non-user for the rest of their time at Cambridge.

For this reason I think it’s still important to have the initial contact of the induction tour and for students to have a positive impression of the library at this stage. I want to tell them: If you take only one thing away from the induction tour, I hope it’s that we’re here to help you do the best you can. We can then try to make the library experience easier through good design, through helpful staff and through ongoing training opportunities that emphasise what the library can do for users, "induction" will be an ongoing process of mutual development.

So I'm looking at a three-facet approach to inductions: the tour (Fresher's week), ongoing information/library literacy training and good design of library services and informational material. But how to make that initial tour inviting and engaging? Part of it is just being a friendly face, but I must admit I'm really tempted by the idea of a game or other hands on activity. But because we never know how many will be in a tour, I think it's likely we'd have to have people sign up if we tried to "gameify" the induction tours. It's something I'll definitely have to think about. At the end of the day, though, if students leave the induction tour feeling like the library staff are on their side and that it's not a scary place I will consider it a job well done.

What creative ideas have you heard for library inductions? Do you think they're useful or do you think the design of the library should tell users everything they need to know?

Radical librarianship?

I have a lot of thoughts bumping around in my head today, prompted in part by attending some classes and meetings with really amazing, inspiring librarians, but also from reflecting on my job and my place in the vast information landscape. I'm going to try to let some of these ideas out so I'm sorry if this blog post is a bit ad hoc.

I think the place to start with is admitting that libraries, particularly higher education libraries, are part of a system that traditionally reinforces privilege at every level. Although libraries can fill a radical role in communities, that ability lies chiefly in the actions of individual users and librarians. As institutions, libraries are traditionally prescriptive; us (the librarians) telling them (the users) what's good for them and how to get it. To quote nina de jesus, "Libraries as institutions were created not only for a specific ideological purpose but for an ideology that is fundamentally oppressive in nature. As such, the failings of libraries can be re-interpreted not as libraries failing to live up to their ideals and values, but rather as symptoms and evidence of this foundational and oppressive ideology." It is important to identify privilege in order to begin to address the problems it creates in terms of barriers between potential users and information resources.

HE librarians may not be able to rectify the admissions processes and broader economic issues that mean some demographics are underrepresented in our institutions, but we can dismantle our ideas about a hierarchical information landscape in which we sit at the top, or at a choke point between users and information. Many (I would think most) librarians already think this way, but many of the institutions in which we work and the systems in place are inherently conservative and over time the radical politics are driven out of individual librarians by time or my feeling the need to work with the system in order to progress in their careers.

The thing is, institutions and systems don't change unless people make it happen, which is why it is up to us, from the Librarians-with-a-capital-L down to the library assistants like me to struggle to change systems that we think are unfair, outdated or ineffective. I operate under the idea that one should never EVER tell oneself that caring about fairness and access is above (or below) your pay grade. The most revolutionary thing librarians can do is relinquish our power and realize that it was never real. Libraries are at their best in the hands of users. That means our role is to facilitate, to listen, to empathize, to collaborate and to be imaginative more than to instruct, preserve, dictate or control. Our expertise can be deployed in handing over the maps and keys to users and ensuring they understand how to use them rather than clinging onto them and insisting that they follow us.

I've been working with librarians throughout Cambridge lately who are of the same mind, pushing against institutions and ideologies that seem intractable at times to change the face of academic libraries. It's inspiring and humbling to sit around the table with them and have a voice. This encompasses everything I'm trying to do at the moment, from making the rare books and manuscripts collection more accessible to working on creative and engaging ways to help students develop their information literacy to conducting UX studies to find out just how much we don't know about our users. I know a lot of this is old news to lots of librarians, but privilege doesn't go away as soon as you embrace a user-focused approach. We need to constantly check ourselves to make sure we are not making assumptions about our users, what they know and how they work. We especially need to question our assumptions about our role in our users' lives. I believe that the more we hold on to the role of gatekeeper, the more irrelevant we will become. Conversely, the more we empower users, the more central to their academic and social growth we will become.

Anyway, these are the things I've been contemplating lately, sitting at the desk all quiet and everything.

Penumbra shadowing experience at the English Faculty Library

I was lucky enough to do a job shadowing afternoon at the English Faculty Library with Libby Tilley and Helen Murphy on Thursday the 8th of May. The shadowing experience was focused on user education, at my request, and so primarily involved sitting in on a training session entitled “Quickstart for Part 1 dissertations: processes and resources”. This built on a session held in Lent term on building bibliographies using Zotero, so knowledge of this was useful as a prerequisite, though they did cover it again for those who did not attend. The rest of my time at EFL was spent at Issue Desk and in conversation with Helen and Nora, a work experience employee about libraries in Cambridge, the differences in our libraries etc.

The training session

The EFL has a small but very nice IT training suite on site, and this was where the training session was held. There are laptops at each station and the person at the lectern is able to switch from projecting their screen to any of the laptop screens. The laptops can be locked away beneath the surface of the desk when the room is not in use.

The session was a quick overview of the process of planning and writing a dissertation, targeted at first year English students. Its purpose was to allay fears and also introduce students to some of the resources available, particularly those provided by the library. It was based on a hand-out that divided dissertation writing into 5 steps or sections, and each section had a “hook” or useful tidbit that I think students will find memorable when it comes time to start writing. Everything covered was on the hand-out and the services or resources provided by the library are highlighted in a different colour so that students know what the EFL provides. Furthermore, students were asked which section they thought they needed more help on. Responses were a little timid, I think possibly because no one had thought that far ahead yet, so all of the sections were covered.

SECTION 1: Choosing a topic

First, Libby emphasized that in the dissertation the library could help, but that students’ directors of studies/supervisors are gods as far as the dissertation is concerned. However, she pointed out that the first port of call for English students is the “Guidance on dissertations” on the faculty intranet. She recommended reading it over the holidays, since it’s quite comprehensive, and noted that in October there will be an additional and more comprehensive guide to referencing. The second resource she pointed to was a folder of past dissertation topics, which is available to use in the library and can be borrowed from the Issue Desk.

SECTION 2: Planning for effective working

Post-it note exercise - Everyone was handed a post-it note and asked to write one effective working habit they have. A couple of brave volunteers said theirs out loud, but the rest were collected and will be collated and passed on as advice to next year’s first years, so that they are getting advice from their peers rather than tutors, librarians etc.

Libby recommended starting early in case there are niche, hard-to-get resources, but emphasised that the library could get these resources for them if they were given enough warning. Additionally, the library is able to help by performing a literature search. They provide postcards that students fill in with their names and dissertation topics and hand in to the library if they are stuck, if they don’t know where to start, etc. This doesn’t take much staff time but sounds like a really big help to the students. The theory behind having a postcard rather than just giving out an email address is that there is a physical reminder of the service, and because you just fill in the back of the card advertising it and drop it back at the library, it’s a pretty user-friendly way of going about it.

Next, they did a refresher on Zotero, a free piece of referencing software that lives in your browser and recognizes bibliographical information, saves it, and automatically generates bib. references in your chosen format. The final useful advice on planning that was handed on had to do with time management and managing due dates from different libraries through the libraries@cambridge iCal app. (We could easily link to this from the web page and point it out to students who tend to get late fees.)

SECTIONS 3 and 4: Primary Reading and Secondary Reading

Most people indicated that they felt pretty confident with these aspects of dissertation research, but it was interesting to note that most of them had only one or two ports of call and did not cast their nets wider. Also they didn’t know the drawbacks of the databases they used (namely JSTOR and its out of date information, and the fact that many people assume that just because it’s from JSTOR it’s “solid”, so they don’t have to think critically about it). Of the sources listed on the handout, people seemed not to have heard of a few of them, and hadn’t used most of them.

They were given a chance to play around in some of the databases using a topic chosen from a list. This let people get more familiar with things like Project MUSE and LION, which have complementary coverage to what JSTOR provides. They also pointed out the UL’s uncatalogued book search function, which allows you to search for books by ISBN, request they’re moved to the front of the queue, and then have them put on hold for you. Because most people said they felt comfortable in this area, they did not cover every idea listed, but the handout would provide further guidance if people wanted it when it came to doing their literature searches, and it seemed like most people picked up on something new they could use.

SECTION 5: Writing and referencing

Libby once again emphasised the importance of the DoS/supervisor in this process, but pointed out some additional resources and tips, such as Transkills for English and Scrivener.

Overall impressions:

The teaching session worked really well. It never seemed to lag and it wasn’t too overwhelming. I think each section had a memorable hook that students will have come away with, and it was very nicely broken up with activities. I think it’s especially useful giving people the time and encouragement to try at least one new database on their own rather than doing a demo on a projector screen and mandating what they try. The EFL website is also really sensibly laid out with resources grouped by subject and then overarching, key resources like LION, Zetoc, Project Gutenburg, etc. down the side (grouped by primary and secondary resources). The single handout means that students weren’t sent away with loads to read so I think that was a really manageable amount of information overall, and the main starting point, the “Guidelines for Dissertations”, was reiterated at the end so people were reminded where to go first.

Other things I noticed about the EFL

- Duplication of knowledge and abilities: EFL tries to make sure that there is nothing that only one person knows how to do. This makes people’s roles more diverse and ensures that if someone leaves or is on holiday things get done. - Issue Desk entails other duties when the circulation slows down: Made possible by the fact that most users come just before or after lectures so the rest of the time is fairly quiet. Duties include checking the library email and sorting messages into relevant folders, book labelling, book repairs etc. - Issuing is done with RFID tag readers and there is self-issue as well as staffed issue desk, but books issued at the staffed one are still stamped out and in, which slows the process down. - Tea@3, poetry wall, relaxed rules about bags, comfy seats etc. make it a very comfortable library to use, people seem happy to approach library staff. Issue desk is low down, makes it look approachable. (Side note: I hate our Issue Desk. It’s like a fortress of doom.)

What I’ve taken away from the experience

First and foremost I’d say that it’s always worth visiting other libraries. It gets you to question the way you do things, generates ideas and promotes the sharing of knowledge and resources between libraries, which can only be good for the users of those libraries.

Second, I am very interested in and impressed by the reference help that the EFL provides. Not only are there clever and user-friendly schemes like the dissertation topic postcards, but the entire culture and design of the library promotes the impression that the staff are there to support learning, not as an obstacle to it. I think TCL needs to realise that whatever our sentiments about reader services may be, we do not give off a friendly and supportive impression and that should certainly be addressed.

Third, I feel like there is the potential for us to provide support in the form of user education initiatives. They would certainly look different from those at Faculty libraries, as they would not be subject specific, but we could have closer relationships with the directors of studies, point students toward the right sources of information and target our information toward the specific assessments that students of the College are doing. I will brainstorm ways of doing this. If we don’t have space to do it in person I think that we could develop something good online.

If nothing else, I think we should focus on structuring the library to be more approachable and develop some form of reference service, especially in subjects that do not receive that kind of support from their Faculty libraries, and on working more closely with Directors of Studies in the College.

Many thanks to Libby and Helen for letting me sit in/chat with you guys about library things!