Student supervision and feedback

Why strong labs sometimes submit weak papers (Josh Schimel)

I’m sure almost all of us have had to deal with manuscripts where we knew it would be much easier to take the data and just write the paper ourselves, rather than try to coax a student’s work into a polished form. But doing that would undermine them; they need to learn how to write good papers, how to manage the process, and how to gauge when a paper is ready to submit.

In Writing Science, I pointed out that “doing science is inherently an act of both confidence and humility” and that getting the balance between them “is one of the greatest challenges all developing scientists face.” Learning that balance involves both over-shoots and under-shoots. For a student to become a fully fledged professional and peer (as they should), they need to establish ability and confidence, and to develop an independent identity. They need room to grow and to become a peer.

I’ve sometimes found myself in the situation described above, where the student work I’vwe been going through has seemed to have so many issues that it would be easier for me to simply rewrite the work than to try and guide the student towards deeper understanding. I have to fight the urge to correct what I see as problems, and give feedback that aims to help the student see the problems themselves. Until they learn how to recognise the issues and make appropriate changes, we’re not helping them to develop as academics and colleagues.

As a supervisor, one of the ways that I’ve found to avoid this has been to switch off the “Track changes” feature. While it gives me a sense of satisfaction to see how much I’ve “helped”, I know that the student will probably just “Accept all changes”. This means that my attempt to show them a “better” way has missed the point because they won’t be paying attention to it anyway. I was alerted to this fact when the due date for a Masters proposal was rapidly approaching and the student was beginning to stress about their impending submission. They were concerned when I suggested that they begin final preparations, and responded that it wasn’t ready yet, because I, the supervisor, had not yet corrected all of the grammar and spelling. That’s when I knew that I was the problem.

Now when I give feedback I try to ensure that my comments are in the form of questions that highlight what I think are gaps in the students thinking or writing processes. I try to give suggestions for actions that the student can take in order to addresse these gaps, and sometimes offer links to resources that they can use. The point is that it is the responsibility of the student to take action, based on the feedback, in order to improve the work.

If basic research were conducted under the conditions of educational research

Council of the rats, from Wikimedia Commons
Council of the rats, from Wikimedia Commons.

This post was originally published at /usr/space.

Basic, bench research study – you are testing the mechanism of an airborne viral infection on lung function:

  • You have a line of carefully bred rats, all genetically identical.
  • You keep them under controlled conditions of temperature, food, exposure to the environment and isolation from other rats.
  • You expose them to the virus under conditions to ensure they get identical levels of exposure to the pathogen – viral concentrations and durations of exposure.
  • If desired, you expose them to the virus multiple times at specified intervals.
  • After an appropriate interval, you sacrifice the rats to examine the lung tissue for evidence of the effect of the virus.

If you take the same kind of study and try to implement it under the conditions of most educational research, you have something like the following:

  • Your rats come from everywhere: white rats, sewer rats, pet rats, roof rats, Norwegian rats, and even a few mice. In fact, the rats are INTENTIONALLY selected to be diverse, rather than uniform.
  • You have no control over where the rats live, what they eat, what they do, what other rats they consort with, or what activities they pursue.
  • You expose them to the airborne virus in a large room when all the rats are gathered together by releasing the aerosol at the front of the room and letting if diffuse through the rest of the room. During this exposure interval, some rats come in late, some leave early, some are sleeping and thereby breathe in less of the virus, while others are active and breathe in more. Of course, some of the rats aren’t even there.
  • If you want to have multiple exposures, some of the rats from the first exposure will now be absent, whereas other rats will be there for the first time.
  • After exposure, many rats intentionally try to share the virus with their fellow rats.
  • At the same time, dozens of other researchers are using the same rats for their own studies, exposing them to various agents, running them through various mazes, observing their behaviours and feeding them all manner of diets.
  • Instead of holding them in controlled conditions while the virus establishes itself, you have to release them back into the wild, where they roam freely, engaging in all sorts of unexpected activities and exposing themselves to all sorts of other viruses.
  • When it comes time to perform the autopsies to examine the effects of the virus, you first have to catch as many of the rats as possible. Some evade capture and other that you trap don’t look familiar to you and you question whether they are really part of the study.
  • Then, you find that the ethics board denies you the opportunity to sacrifice the rats. Instead, you must develop tests to infer the effects of the virus or questionnaires to ask the rats “how they feel”.

Larry D. Gruppen (PhD), University of Michigan Medical School

Writing a research proposal for T&L

Here are my notes from a presentation by Prof. Denise Wood on developing a research proposal for projects looking at T&L.

Image from Tony Duckle's Flickr photostreamUnderstanding the funding body is important when it comes to applying for funding. Disciplinary specific proposals may not be successful when it comes to T&L projects.

Local evidence of successful projects is important before applying for larger grants. Collaborative teamwork is a great way to build ideas and test concepts. Local resources help you get started and build a track record. Generating pilot data helps to begin publishing. When panels review research proposals, your previous experience in obtaining funding and successful proposals is highly emphasised.

Why are you undertaking the study? Knowing your goals will justify your design decisions. What are your goals:

  • Personal
  • Practical
  • Intellectual / theoretical

Writing proposals is closely tied to career trajectory

How are you using research and research projects to improve your teaching practice?

What conceptual framework are you using:

  • Research paradigm
  • Experiential knowledge
  • Existing theory and research
  • Pilot and exploratory studies

Interested in addressing a gap, bringing in personal reflections that guide and influence the research. If you only think of your conceptual framework as a literature review, then you limit the scope of your research to what others have done.

Research questions:

  • What is the relationship between the goals and the conceptual framework?
  • Help to guide the actual research design / methods
  • Used to connect the problem and practical concerns
  • Should be specific and focused on the study
  • Need to allow flexibility to reveal unanticipated phenomena (if the questions are too focused you may miss emergent ideas)
  • Need to avoid inherent assumptions as they bias the study

Find a balance in the number of questions (3-4 is usually adequate)

Begin with divergent thinking to allow yourself space to explore many possibilities. Mind mapping is useful to identify high-level ideas. Begin reading broadly and then begin narrowing the focus. You can’t answer all possible questions in one study.

Try to avoid getting too caught up in the details of the research methods. Only use methods that you understand.

Note that you will be informed by your own epistemological understanding of what knowledge is and how we come to know. Your methods (quantitative, qualitative or mixed methods)  will most likely mirror your understanding of how we come to know. This will in turn guide how you sample, gather and analyse data.

For local studies, it’s OK to use a pilot within a classroom. Use this to identify a single context. Larger proposals would be better to expand the scope of the study and test the outcome of the pilot. On the basis of the smaller studies, you can make an argument for the larger study. Think laterally about how you can collect data.

  • Validity: How might you be wrong?
  • Bias (what assumptions do you bring with you? Results and interpretation distorted by your own values and preconceptions)
  • Reactivity (quantitative researcher may try to control for the effect of the researcher influence; qualitative researcher looks at how they actually influence the outcomes)

How do you reduce bias and reactivity?

  • Studies should be intensive and long-term (not the same as longitudinal study)
  • Gather rich, thick data (less likely to get from surveys / questionnaires; rather use interviews or focus groups)
  • Respondent validation of outcomes (is what you heard the same as what they meant?)
  • Identifying discrepant cases or evidence (you should take outliers into account, but identify and reflect on them, not necessarily include in the main data and suggest reasons for the discrepancy)
  • Triangulation
  • Comparative data (look at different contexts and populations)

Proposal checklist:

  • Identify a funding body
  • Objectives of the funding body
  • Use the guidelines that the funding body provides
  • Previous funded research and see what has been accepted and / or rejected
  • Links with existing research that the body is involved with
  • Evidence of value, need and benefits (institutional, local, national, international)
  • Background / conceptual framework
  • Methodology
  • Evaluation strategies are valued in educational research
  • Engaged dissemination whereby you share your results as you go, using a variety of methods, including publications, conference presentations, social media and workshops
  • Budget: must meet funding body requirements, realistic, value for money, justify costs
  • Milestones: linked to objectives and outcomes
  • Researcher capabilities: ensure you can deliver what you say you can, track record, previous collaboration, strategic, roles and responsibilities, realistic within workload

Try to model your proposal on successful projects. Learn from the mistakes of others. Sit down with a colleague and ask for constructive feedback.

Explicitly make reference to important and contextually relevant policy documents.

Identify how your research is going to create systemic change.

How are you going to evaluate your process and outcomes?

  • Formative: should be ongoing and used to modify project
  • Summative: can be broad and can go beyond the stated outcomes

Design-based research: can use milestones that are linked to formative evaluation. Identify problems early on and adapt quickly.

How are you going to convince the funding body that the people you’re collaborating with are adding value to the project? You must justify the presence of every team member and highlight how they will contribute.

How are you plugging the holes that funding assessors are going to be looking for?

Differentiate between deliverables (the tangible products that will come from the project) and outcomes (the achievement of stated aims and objectives).

This was originally posted at /usr/space.

UWC writing workshop

Image from Wikipedia article on writing
Image from Wikipedia article on Writing

Last week UWC hosted a writing workshop for academics. I always enjoy writing workshops because there’s always something I find in them that makes me think about my own process a little bit more. I wasn’t able to attend the full sessions every day, but managed to make a few notes while I was there.

Writing an Introduction

CARS model – Create A Research Space (John Swales). Using the Introduction as a way of creating a space, or a context that the research will take place in. Publishing as a way of creating a conversation in a field. I write the Introduction last, as a roadmap for the reader rather than as a roadmap for my writing. I can’t change the Method and Results, so I use those sections as anchors and go back to the Introduction to make sure that there’s alignment between the major sections.

Genre – ways that members of a community agree that certain discourses within that community will be presented

Writing as a process of learning about what you’re writing about. The process changes you and changes itself as it moves forward. Sometimes you don’t really know where you’re going or how you’re going to get there, but the act of writing creates can create a pathway from where you are to where you need to be.

Try to avoid:

  • Repetition
  • Unnecessary background and context
  • Exaggerating the importance of the work
  • Weak statements
  • Assumption that the reader knows where the research is located i.e. context is lacking
  • Not focusing on a clear and compelling research question i.e. it is too broad

Establish a niche by showing that previous research is not complete (can be negative evaluation of other studies, or positive justification for your own)

Using the Introduction as a way to establish authority and situating yourself and your own work within a context.

Knowing the steps in this model can be used to give feedback to other people’s writing. The tacit understanding of good writing can be made explicit. Is also a good way to actually read articles.

Keeping track of sources and versions as part of the Literature Review

Three uses of sources

  1. Making notes: Help formulate the question; Quick read to spark interest; Recording general ideas
  2. Reading for an argument: Help to make a logical argument; Look for similar arguments to what you want to make i.e. a logic checklist; Use the literature to make points that you want to address; Turn major points into questions that you want to answer
  3. Reading for evidence: Most common reason but not necessarily the most important; Reporting evidence completely and accurately, cite source carefully; Try to locate the original source; Don’t try to collect everything (we often feel a need to gather and read as many sources as possible. Rather try to be selective about a few good sources)

Preserve what you find. Record bibliographic information and notes accurately. When taking notes from literature, summarise the main points, highlight issues/data/methods that are important. Identify when you quote, when you paraphrase and when your notes trigger a new thought.

Share you resources with colleagues. Discuss what you’re reading with others, and discuss why what you’re reading is useful (or not). How does your reading shift or shape your topic or questions. Use sources to revise your question and topic (check with your supervisor).

Saving versions

  • Understand the concept of versioning
  • Use file sharing rather than email e.g. Dropbox
  • Rename files with dates, supervisor names
  • Keep an archive folder with older drafts (don’t delete anything)

Imagination and writing creatively

I sometimes feel like not writing because it’s like I have nothing to say. Sometimes you  who don’t write because the ideas aren’t there but it’s important to understand that ideas don’t emerge from nothing. Being embedded within conversations is one way for the seeds of ideas to be planted. Over time, and over many conversations, the seeds begin to grow and you feel like you have all these ideas that came from nowhere, overnight. As a writer, it’s important to keep track of all of these seeds and add to them over time. For example, keeping drafts of blog posts with links back to original sources, or keeping notes of books being read, or just keeping a short audio note reminding yourself that “this is an interesting idea”.

How do you deal with having too many ideas?

Think of note-taking and reading as a critical conversation with the author, rather than just reading and making notes. Being critical of the writer / writing style open us to further avenues for thinking about the topic.

What is the purpose of the review? This influences how you approach it.

  • Find published research
  • Read and skim: identify major points, important contributors
  • Map the literature: outline the major topic with sub-topics, highlight themes, narrow the boundaries of what to focus on
  • Read and critique smaller portions to identify exemplars
  • Write a focused review that introduces the perspective you want to use

Writing is not a linear process.

Think about beginning locally and then moving globally. Usually we write to narrow the focus e.g. by beginning globally and moving locally.

Data analysis and reporting

What are some of the challenges you experience with data analysis and reporting of results? I find it difficult to only use the data. There are experiences that exist outside of the data, tacit knowledge, that I find difficult to integrate with the “formal” data.

Avoid simply listing quotes under categories, try to create links between the major concepts.

Data analysis is shaped by the research methods you choose.

Depending on whether the data is quantitative or qualitative, the presentation will be different.

Ensure that the discussion of the data goes further than simply repeating the results. The role of the author is to interpret the data in the context of the literature, to go further and unpack what can be inferred. Acknowledge that analysis is an act of creation, informed by personal beliefs and biases. We can try to reduce bias but should be aware that how we interpret the data can’t really be separated from ourselves.

Discussion of findings, conclusion, submission and peer review

Go back to the beginning. Review the research question. Remember that the process is not linear.

In quantitative research, it makes sense to separate out the results and discussion of those results. This isn’t the case in qualitative research where the discussion of the results are usually best done together.

Make sure that themes are logical and that they build on each other. These themes should also speak to the key questions and ideas that were presented in the Introduction.

Unpack the quotes and narratives clearly. In qualitative research, avoid using too many quotes to illustrate the same point.

The presentation of the findings should follow a logical flow. Guide the reader through the piece so that they feel like they are moving through a process. Try to integrate the results into a narrative.

Consider using pseudonyms rather than “Participant 1” in order to avoid objectifying them. You could even ask participants to provide their own pseudonyms.

Conclusion: Here is what I did, what I found, what it means and some things that might happen next.

Finalising the document: check on content and alignment. Is the context and rationale clearly outlined? Are the questions or problems clearly stated? Is appropriate and comprehensive literature reviewed? Are you joining the conversation with familiarity? Have you outlined your methodology clearly and in appropriate detail? Have you summarised, concluded and drawn out key contributions of your work? Are your references in the correct format? Do you need to acknowledge anyone? Is the work formatted according to the journal requirements?


Getting rejected

Seven reasons why journals reject papers (Pat Thomson).

We all get rejected by reviewers and editors at some point (some of us more than others) but that doesn’t make it any easier. The trick is to write intentionally with the aim of trying to avoid that heartbreaking email telling you that your paper was great, it just wasn’t quite right. The link above gives a short summary of seven common reasons that papers get rejected by journals. Make sure that you don’t make these mistakes the next time you submit.