During the last two years of undergrad and the first year of my M.S.E. program, I've come to value the habit of reading academic papers consistently. My prior motivation to read was solely driven by fulfilling the homework requirements for paper-reviewing seminar courses, which are popular at the graduate school level. Across these classes, I was exposed to a variety of fill-in-the-section reading templates. At the time, I didn't think much of the template layouts.

My motivation and attitude towards paper reading changed one semester into my Master's program. My recent interest in working on novel research ideas with the Princeton NLP group has ignited a new desire to read papers to close knowledge gaps in areas where I'm under-informed (in hindsight, this is a very obvious motivation, but it didn't mean that much to me until now 😝). However, I soon realized, I don't actually know how to read a paper. Now that I was reading papers during unstructured free time, without the framework of class-required reading templates, many meta-questions regarding the paper reading process began to surface.

My first observation was that I needed to read a paper with purpose. My main current focus is thinking about how to use web interactions to design language agents capable of completing real world tasks (Edit Aug 2022: resulted in the WebShop project). Initially, I was handed a variety of suggested related work, and I'd often read deeply and thoroughly. This approach seems very diligent at first glance, but in reality made for some very frustrating reading experiences. As an inexperienced reader, I didn't know what to look for or pay attention to; the distribution of the importance of any paper's content appeared very flat to me because I had a "reading just to know" attitude. When I mentally surfaced and took a step back while reading a paper, these questions came up frequently:

1. What's the point of reading this? The motive should be deeper than "this paper is in <general area of interest>".
2. I come across a referenced concept that I'm unfamiliar with -> do I read "in" or read "over" (debugging reference just in case it wasn't apparent 😉) the concept?
3. Where does this paper fit in (to my mental map)? So I understand the paper - good! But how does their work inform my own thoughts?

It was clear at this point that my fledgling "know just to know" attitude not only made paper reading tedious, but also limited the depth of engagement I could have with my reading. It also became apparent that to carry out reading a paper with purpose, I needed to track my thoughts in a structured fashion - something that would be difficult for a novice like me to pull off without writing.

Hence, the template comes in to play! To ingrain the point of reading with purpose, I decided that a good exercise would be to come up with my own questions. I wanted these questions to encourage me to capture 1. why I was spending my time on this paper and 2. what about this paper addresses my goals.

I came across Prof. Srinivasan Keshav's article titled How to read a paper, which I highly recommend. For me, this paper not only resonated with many of my struggles, but also proposed a paper reading technique that worked well for me.

The key idea is that you should read the paper in up to three passes, instead of starting at the beginning and plowing your way to the end. Each pass accomplishes specific goals and builds upon the previous pass: The first pass gives you a general idea about the paper. The second pass lets you grasp the paper’s content, but not its details. The third pass helps you understand the paper in depth.

What I very much like about this method is that going through all three passes is not necessary. For instance, he discusses with respect to the first pass that:

Using this information, you may choose not to read further. This could be because the paper doesn’t interest you, or you don’t know enough about the area to understand the paper, or that the authors make invalid assumptions.

There's an implied decision on how purposefully one wants to read the paper. Each subsequent pass is a personally justified and intentional step from an initial, holistic take towards valuable details, assuming the reader decides that such details are worth understanding. There is no need to read for the sake of reading.

I used Prof. Keshav's paper as a set of stakes for choosing my questions around. I looked at a couple of templates from past courses and online resources and chose questions that:

1. Engage my reason(s) for reading this paper with the content.
2. Can be extended upon / answered with greater detail if I choose to make multiple passes. If I choose to read this paper again in a month, this question should allow me to recollect and build on thoughts from the previous pass.
3. Helps me generate action items (i.e. new papers to read, an experiment to try)

I found this StackExchange with a similar question and these templates [1, 2, 3] quite helpful.

I think what was important for me was not that these are the universal "best" set of questions (they are not). It's more that they are worthwhile to me and encourage me to invest in and thinking deeply about what the author has to say. The end result is a several of cliche but necessary regurgitation questions (in my own words, restate what the paper says), sprinkled with a couple questions that target what kind of role the paper plays in my own train of thought.

Here it is:

1. Why this paper?: Why did I chose to read this paper? The source may be straightforward (i.e. class paper, professor recommendation) or more personally derived (i.e. arXiv surfing, someone's reading list). I'd like myself to add another line describing what I'd like to gain more insight on via this paper.
2. Context: How did we get to this problem? In practice, this will likely be heavily drawn from paper's related work section, but my own goal is to remind myself how this problem ties into other research. What is/was the state of affairs at the time of the paper's publication, and what gap is this paper zoning in on?
3. Problem Statement: What is the problem? What's the contribution?
4. Notes: Methods and results that stood out to me.
5. Questions: What did I not understand? Will I address these gaps? If so, how? Also, what did I not understand initially, but then resolved? How'd I arrive at the resolution?
6. Looking Forward: Take aways and free space for any additional thoughts. What are my conclusions, if any? Anything I can look into that might be worth trying?

I'll do my best to abide by this questions for summarizations of my future readings (that you can find in the Readings tab!).

Thanks for reading! I'm curious to here from you, what questions or approaches do you keep in mind or exercise when reading?

Edits

11/3 - Came across this Twitter take on how to approach paper reading and liked it! Purpose-driven reading is the way to go.