Design, Implementation, and Evaluation of an Online Manual For Chemistry 32

LDT Master's Project

June 1st, 2000

By Kalée Gregory

Abstract

My LDT Master's Project is the design of a website learning environment for Stanford accelerated course Chemistry 32: The Frontiers of Chemical Science. Chem 32 is offered only during the fall quarter and is an introduction to modern chemical research in the various sub-fields of chemistry (organic, inorganic, bioinorganic, catalysis, physical, analytical). It is aimed primarily at first-year, advanced placement undergraduates seeking a taste of what is in store should they decide to major in chemistry. I was a teaching assistant for the course in the fall of 1996 and head teaching assistant in the fall of 1997, and am therefore familiar with the resources available and problems that students typically encounter during the course.

I designed, implemented, and extensively user-tested my website over the first two quarters (summer and fall) of LDT. User-testing involved interviews and conversations with the students and current TA's as well as two class-wide questionnaires tracking Chem 32 resource usage and study patterns. The remainder of this project entailed implementing various suggestions from the students and TA's.

The website is at http://kalee.tock.com/chem32.

Outline of this document (click to skip to the corresponding section):

I. Background of the Problem
   A. Chem 32 Pedagogy
   B. The Students

II. Learning Problems Addressed By This Project
   A. Three-dimensional Visualization
   B. Reading Versus Practice
   C. Combining Problems and Solutions
   D. Passive Versus Active Problem Solving

III. Approach
   A. Design Philosophy
   B. Project History
      1. Summer quarter 1999: Chem 32 Website Design and Implementation
      2. Fall quarter 1999: Design Studies
         a. Interviews
         b. Questionnaires
      3. Winter quarter 2000: Redesign
         a. Priority #1: Anchored Texts
         b. Other Redesign Ideas
      4. Spring quarter 2000: Redesign

IV. Literature Review (References)
V. Appendices
   A. Interview Transcripts
   B. Questionnaire Data
      1. November 3rd Survey and Data
      2. November 19th Survey and Data

I. Background of the Problem

Chem 32 is a Stanford course designed to introduce the advanced introductory chemistry student to modern research. Conceived of and taught by Professors James P. Collman and Richard N. Zare, it is notorious among Stanford undergraduates as "the most difficult course on campus" because of its pace and the vast amount of material that is covered. Professor Zare compares the course to "drinking from a firehose," an apt description for the vast quantities of chemical information and problem-solving techniques that students must learn.

There is no textbook for the course; Chem 32 is unique enough that no existing text would suit. Students who take it are assumed to have a firm understanding of introductory college chemistry and to be intellectually ready to grapple with actual modern chemical problems. Therefore, standard introductory chemistry texts would not suffice and more advanced chemistry texts are too specialized. In place of a textbook, Chem 32 students buy a course reader, which is a collection of Jim and Dick's lecture slides, and a lab manual, which outlines the experiments for the course. The course reader can be incomprehensible to the student who does not attend lecture, as explanations to accompany the notes and figures are often provided verbally in class. My mission was to provide a much needed content resource that would take the place of a text.

A. Chem 32 Pedagogy

The students who take Chem 32 are extremely gifted; almost all have the highest possible scores on the Advanced Placement chemistry and calculus exams. The Cognition and Technology Group at Vanderbilt has undertaken extensive exploration of the nature of such expertise. "A major conclusion from the research," they write, "is that high-level expertise requires a great deal of domain-specific knowledge . . . the development of thinking is limited more by lack of knowledge than by the absence of general logical capacities." (CTCV) Professor Collman addressed a similar point in a more down-to-earth way in his second Chem 32 lecture of Fall, 1999. "We will try to limit the facts we expect you to know to what's important," he told the lecture hall full of students. "But you will have to do a fair amount of memorizing. We're training you to be chemists, and you can't go into a chemistry lecture with a load of books and encyclopedias and expect to know what's going on.

In this Professor Collman touched on a very important point about science education. The goal of teaching students to solve problems instead of having them internalize information, though embraced by many non-science educators, is inadequate for the training of scientists and thus is not the pedagogy employed in Chem 32. In order to solve real scientific problems, especially in creative ways that have not been tried before, one needs to have quite a lot of information internalized and at one's immediate command. Such problem-solving can be likened to working a jigsaw puzzle, except that appropriate combinations of puzzle pieces are not physically available on the table but must be identified and combined in one's mind. As Andrew Molnar puts it, "Often the difficulty in solving the problem is not inherent in the nature of the problem, but in the tools that are available to us." (Molnar, 1997)

B. The Students

The highly gifted nature of the students who enroll in the Chem 32 is problematic for the would-be educator because gifted students have such highly individualized learning styles and study patterns. Stanford Professor Patrick Suppes has made a thorough study of such individual differences in gifted math and science students learning with technology. He consistently finds a difference of a factor of two in many measurable aspects of their learning styles and outcomes. (Stillinger and Suppes, 1996 Cope and Suppes, in preparation). These students have highly variable learning needs.

I wanted to create an environment in which Chem 32 students could obtain much-needed individualized support for their studies during the course. In this environment, students should have access to practice problems organized by topic, their explained solutions, and a full explication of the concepts covered in the course. I decided that it should be based online for a number of reasons. First and most practically, I was concerned that students already have to buy so many supplies (course reader, lab manual, lab notebook, etc. etc.) for Chem 32 that the addition of yet another item to be purchased would engender student resentment. Jim Collman shared my concern. We decided that the best way to get around this problem would be to put the materials on the web so that anyone in the course could access them for free. Thus, the Chem 32 Virtual Manual was born.

Another reason for having my environment be web-based was that since it was conceived and written in such a short time, it was important to me to be able to edit the text dynamically, incorporating changes as students and TAs find problems and make suggestions. When one student finds a problem (a confusing or incorrect explanation, a buggy link, etc.) I wanted be able to fix it for other students before they encountered it. Paper texts are not amenable to this kind of dynamic editing.

II. Learning Problems Addressed By This Project

A. Three-dimensional Visualization

Three-dimensional visualization is a critical skill for many disciplines, including engineering, mathematics, and art as well as the natural sciences. Chem 32 is a course that relies heavily on such skills. Symmetry and chirality are two of the topics that are explored in depth, and for the explication of these subjects, two-dimensional representations often mislead new learners. In designing my website, I made use of various web technologies to show specific molecules moving and rotating; I also made interactive models that the students could move and rotate themselves. Where these occur, I contrast various two-dimensional representations of the same molecules and phenomena to augment the students' familiarity with conventional chemical notation. This technique has a dual goal; exposing students to multiple representations, some unfamiliar, has the added benefit of facilitating more careful interpretation and thereby deepening understanding. (diSessa, 1999)

The value of such computer-based models as an aid for helping students learn concepts in chemistry is well known by chemistry educators. (Aduldeka et al., 1991; Campanario et. al, 1994; Moore, 1995) Barnea et. al. find that computerized chemical models can be even more effective than physical models due to the rigidity of the latter and students' misinterpretations of the manipulatives as accurate representations rather than incomplete aids to visualization. (Barnea et. al., 1999)

B. Reading Versus Practice

As I set to work collecting and organizing problems from past exams and from my old sections, yet another affordance of the online text occurred to me. I wanted my manual to be primarily composed of problems and solutions. Students typically like to read text before attempting problems, and what most often results is a passive form of studying that is inefficient when it comes to really internalizing the concepts. I believe that the best way to learn science is exactly the opposite: trying problems and consulting text only when necessary to figure out a problem. This forces students to become active learners and to read text efficiently, picking out the relevant points and discarding the others. It is an approach that will serve them well throughout their scientific careers. Inspired by this belief, I actively tried to minimize the text and maximize opportunity for practice problems.

C. Combining Problems and Solutions

In traditional texts, the problem has to be separated from the solution by pages. Most texts publish a separate manual for the solutions to problems, and paper conservation principles dictate that it usually is not feasible to reproduce the problem next to its solution. On the Stanford ChemWeb server, old exams and their solutions are available for students, but the solutions are typed separately from the problems and upon close inspection, do not always match. It has been my experience that problems are more confusing when they are separated in this way from their solutions. In my work, reproducing the original problem with its accompanying images on the solution page does not waste paper because no paper is involved. Edward Tufte notes that "Words and pictures belong together, genuinely together," (Tufte, 1990) and I found his philosophy to be appropriate here.

D. Passive Versus Active Problem Solving

One major design decision that I made early on was not to give the students access to one big document containing all of the problems with their solutions for a given section. If such a document were made available, the temptation for students to read all of the problems and solutions within a topic passively, without attempting to work the problems themselves, would be too great. The solution that I came up with was to provide a document containing the problems alone, with each one linked separately to its solution. In this way, students have access to immediate feedback when they want to check their solution to a problem, but it is impossible for them to read through all of the problems and solutions without interruption. Although the barrier to passive studying is still far from insurmountable, the extra step may make it more likely that students will actually think about a problem before looking at its solution, thereby inducing deeper cognitive processing. One can always hope.

III. Approach

A. Design Philosophy

My design process has been and continues to be similar to that followed at the Education Program for Gifted Youth on Stanford campus, probably because I have spent so much time working there and have come to believe strongly in the philosophy. As Ray Ravalgia, chief designer at EPGY, articulates it: "One of the 'design principles' that we really believe pretty strongly in is not to spend too much time anticipating what students are going to need but instead get a skeletal version together, offer it to some students, see what works and what doesn't work, see what aspects they use and don't use, what their confusions are, and then go back and make improvements, put some more material in, and address those needs." (Ravaglia, 1999) I have done the same here. I pulled together a website designed in a way that I, drawing on my six years of experience teaching chemistry, thought would be useful for the reasons outlined above. While students used the site during the fall, I collected as much feedback and data as possible. During the winter and spring, I will implement one or two of the most pressing new feature suggestions that came out of these design studies.

B. Project History

1. Summer Quarter 1999: Chem 32 Website Design and Implementation

I spent the summer quarter exhaustively working on content; collecting problems, typing up explanations, and getting them online into the desired format. I also managed to write some summary texts, make a few Java applets for interactive pictures of molecules, and put up a simple web database, which I wrote in Perl, to collect comments and suggestions from students.

2. Fall Quarter 1999: Design Studies

During the fall quarter, I attended a fair number of the lectures and cultivated a good relationship with the TA's. The students and TA's often emailed me personally with questions and feedback about the site, alerting me to misleading passages in the text and errata in the problems. Throughout the quarter, I remained committed to fixing problems within 24 hours of receiving email about them. In addition to conversations and emails, I conducted three major design studies: a session of interviews over a week's worth of laboratory periods and two class-wide questionnaires.

a. Interviews (see Appendix 1 for the data)

After the first exam, I came into the students' laboratory periods and did informal interviews to find out whether and how the students had used my website in studying for the first exam. All of these I tape recorded and transcribed for my personal use in deciding which features of the site to augment. See Appendix I for the interview transcripts. Of the interview as a formative assessment technique, Dix et. al write that "Interviews have the advantage that the level of questioning can be varied to suit the context and that the evaluator can probe the user more deeply on interesting issues as they arise," (Dix et. al., 1998) and this was indeed my experience, as reading the transcripts will reveal.

One prominent finding from these interviews was that many of the students were using "Chemweb" in addition to or instead of my site. Chemweb refers to the departmental server that has information for various courses, including back issues of old chem 32 exams. I put together the Chem 32 segment of Chemweb two summers ago, but it was hurriedly (and correspondingly poorly) designed, has not been well-maintained, and has not been changed very much since my involvement. Many of the students mistook the spiel I gave on the first day of class as a spiel for Chemweb.

However, about half of the students had used my site and were very positive about their experiences with it. Many of them thanked me profusely for writing it. Highlighted by the students as particularly good aspects were the organization of questions by subject, the summary texts, the convenience of being able to check answers immediately, the fact that the answers are not on the question page, and the interactive figures (though these were also cited as a time-sink by one student who said he "wasted" hours playing with them). When pressed for suggestions for changes and new features, the students favored anchored texts, more problems not from old exams, a hypertext glossary, and a text section on nomenclature, which I hadn't written partly for lack of time and partly because it is one of my personally most un-favorite subjects.

b. Questionnaires (see Appendix 2 for the data)

The feedback I got from students during the interviews was instructive. In addition to this, however, I felt I needed numerical substantiation of what the students were saying; i.e. more concrete data to analyze on resource usage and study patterns. Despite the fact that surveys are less flexible and consequently less probing than interviews, their ability to be analyzed more rigorously makes them a valuable tool in design studies. (Dix et. al, 1998) Therefore I administered two class-wide questionnaires on the mornings after the second and third Chem 32 exams. See Appendix 2 for the tabulated questionnaire data.

The second of the surveys, which tracked specifically Exam 3 resource usage and study patterns, probed response to the change in the site design for the thermodynamics and kinetics text section. Both of these were topics that were covered on Exam 3, and I had redesigned the corresponding text sections in response to student suggestions from the interviews and the first survey. Unlike the first questionnaire, the second also included a question about the first language of the survey participant. (Inclusion of a question about first language was the result of a consultation with Professor Kamil, in which he noted that extensive reading of texts is often indicative of non-native English speakers-this indeed did prove to be the case.)

The questionnaire data vividly substantiate Patrick Suppes's findings of individual differences in gifted students' learning. (Cope and Suppes, in preparation; Stillinger and Suppes, 1996) Time spent on any one study method did not correlate with success on exams. As a side note, it is interesting that students were more likely to print out exams from Chemweb and less likely to print out problems and solutions from my site. This correlated well with my intentions for the website as a learning environment in which the only paper needed would be for the student's scratch work in doing practice problems.

3. Winter Quarter 2000: New Feature Prioritization and Selected Redesign Implementation

During the winter, I prioritized redesign ideas and implemented a few of them, as described below.

a. Priority #1: Anchored Texts

My first priority is augmenting more of the text sections and providing anchors. This was requested numerous times by both the students and the TA's. In addition, the experimental text section on thermodynamics and kinetics written as a study aid for the third exam in this format garnered extremely positive comments from the students on the questionnaires.

The anchored design format also makes sense within Kozma's model of the expert learner. Kozma writes: "Bringing schemata and purposes to the task, [expert learner] subjects would typically read by scanning rapidly over tables of contents and by using certain words to trigger their attention to question a particular title more actively . . . Having identified an article of interest, they would read parts of it selectively and nonsequentially, jumping back and forth, perhaps reading conclusions then introductions, perhaps scanning figures, and finally reading those sections more carefully that fit their purpose. If an article did not readily fit with their comprehension schemata, the readers would weigh the cost of working through the difficulty against the potential gain relative to their purposes." (Kozma, 1991)

Within this framework, anchored lists of what students have to know at the top of the text sections is valuable study aid. These allow the flexibility of having different students able to focus on the aspects of the material that they are having the most trouble with. After consideration and consultation with a user interface expert, I decided to keep the text for each topic all in one file, headed with a list of anchor tags so that students can immediately see a list of what they need to know and jump to sub-sections on which they feel they need work. This deviates from the often-used "chunked" approach to web design, whereby sub-sections of information are placed in separate files and meta-organized by frames. The justification for my deviation is four-fold. First, via the transition paragraphs between subsections, students can form a richer mental model of how the different facets of information fit together. If a discussion of the steady-state approximation were physically separated from a discussion of finding rate equations, it would be that much easier for students to fail to make the connection that the former is a special case of the latter. Second, students who are studying typically are not searching for specific information, as a researcher would be. Instead, they want an overview of everything they need to know about a given sub-topic, and as such want to at least skim most of the sections, pausing to look more deeply when they encounter less familiar words or material, as Kozma's expert learners did. For students such as these, having to click multiple times to get through all the information is annoying. Third, having all of the information for a given topic in one file allows students to print it out easily. From the interviews and from questionnaires it became apparent that most students had access to free printing and many made use of this in their studying for exams. Fourth and most importantly, the students requested it, and reacted positively to this format in the experimental thermodynamics and kinetics text sections.

b. Other Redesign Ideas

Some other potential redesign ideas are listed below. This list is an un-prioritized conglomeration of functionality that has been on my to-do list from day one, suggestions from the students, and suggestions from Professors Michael Kamil and Decker Walker during private consultations.

  • Videotape or take pictures of class demonstrations to put online.
  • Implement search functions that find where concepts occur in the text and where they occur in the practice problems. Done.
  • Allow students to create their own practice tests by specifying how many randomly selected problems from each topic they want to work on.
  • Make interactive study sheets with concepts linked to more complete explanations so that students can focus on what they particularly need to study (an indexed vocabulary list).
  • Make a hypertext "glossary" for the course.
  • Add a section on nomenclature and other topics that are currently not covered by the text.
  • Add more practice problems that are not from old exams. Done.
  • To encourage yet deeper cognitive processing, increase the (currently one-click) barrier to seeing the answer to a particular question. This could be implemented by requiring them to enter a keyword or phrase on the way to the answer. This could get annoying quickly, so it would require a careful and thoughtful design.
  • Provide a mechanism for learners to record what specific difficulties they are having with particular problems that would support eventual generation of a database of targeted hints and sketches that students could access when stuck.

4. Spring Quarter 2000: Implementation of Redesign Ideas

During the spring, I continued implementing the redesign features as labeled above. I also moved the site to its latest location and changed the color scheme substantially.

IV. Literature Review (References for this Document)

  • Adduldeka, S., P. Akhter, P. Field, P. Nagle, E. O'Sullivan, K, O'Conno, and B. J. Hathaway (1991). "The use of desktop molecular modeler software in the teaching of structural chemistry." Journal of Chemical Education, 68: 576-583.

  • Barnea, Nitza and Yehudit J. Dori (1999). "High-School Chemistry Students' Performance and Gender Differences in a Computerized Molecular Modeling Learning Environment." Journal of Science Education and Technology, 8 (4), pp. 257 - 271.

  • Campanario, J. M., E. Bronchalo, and M. A. Hidalgo. (1994) "An effective approach for teaching intermolecular interactions." Journal of Chemical Education, 71: 761-766.

  • Cope, Eric and Patrick Suppes (in preparation) "Gifted Students' Individual Differences in Calculus and Linear Algebra."

  • DiSessa, Andrea (1999). "Meta-Representation: Native Competence and Targets for Instruction." Graduate School of Education, University of California, Berkeley.

  • Dix, Alan, Janet Finlay, Gregory Abowd, and Russel Beale (1998). "Chapter 11: Evaluation Techniques." In Human-Computer Interaction, 2nd Edition, Prentice Hall.

  • Kozma, Robert (1991). "Learning with media." Review of Educational Research 61(2).

  • Molnar, Andrew (1997). "Computers in Education: A Brief History" at http://www.thejournal.com/magazine/vault/A1681.cfm.

  • Moore, J. W. (1995). Journal of Chemical Education: Software. Journal of Chemical Education, 72(1): 25-26.

  • Norman, Donald (1988). The Design of Everyday Things. Doubleday/Currency Press, New York, 1988.

  • Ravaglia, Ray. Personal interview (1999). Transcribed at http://www.stanford.edu/~kaleeg/ldt/sem/int.

  • Stillinger, Constance and Patrick Suppes (1996). "Gifted Students Individual Differences in Computer-Based Algebra and Pre-calculus Courses." Available from http://epgy.stanford.edu/Other/index.shtml?research.html

  • Tufte, Edward (1990). Envisioning Information, Graphics Press, Connecticut.

  • The Cognition and Technology Group at Vanderbilt. ?Looking at Technology in Context: A Framework for Understanding Technology and Education Research.? In D.V. Berliner and R. C. Caifel (Eds); pp. 801-846. The Handbook of Educational Psychology. New York: MacMillian Publishing.

V. Appendices

A. Interview Transcripts

These are transcripts of informal 'interviews' I did with Chem 32 students while they were doing their labs the week of Oct. 24. I walked around from group to group while they were working and asked them some informal questions, which I tape recorded. For lack of time, I have not reproduced my questions, which always started out with something like: "Hi, I'm Kalée Gregory, I put together the Chem 32 Website at /~kaleeg/chem32. Have you used that site? When did you use it? Did you find it helpful and are there changes or improvements I could make to it to make it more useful for you?" Often the students were heavily involved in detailed experiments that I didn't want to disrupt too much, so I asked a minimum number of questions and moved on. Sometimes, however, I got the opportunity to probe more deeply, asking them among other things whether they printed out material from the site and what they printed out.

Where it seemed appropriate, I transcribed my questions along with the interviews in parentheses. All of the students that I talked to knew that I was tape recording them, that their responses would be used anonymously for improving the site, and that they didn't have to respond.

Summary tally: I ended up talking to about 30 students. 17 students said they had used the site, 15 students said they had not used it. On listening to the interviews afterwards, I concluded that one of the 17 mistook my site for the Chemweb site, so the accurate tally should be about 16 to 16.

  • I've used it to, uh, like the section where it has a part for each chapter--I'd just click on that, and I'd just do those problems, but I wouldn't print it, I'd just write it on paper. And that helped me a lot, that was awesome 'cause I mean yeah, there's a lot of material in the book but there aren't that many problems to actually do, so that helped me a lot. (Me: Are there any ways you can think of to improve the site? Some people were suggesting lists of concepts, etc . . . ) I don't think that would be that big of a deal because it's pretty much in the book--I mean, if it's . . . maybe . . . but I thought that the fact that there were actually problems that you could work out was helpful. The answers were right there too, weren't they? Yeah, that was great. That was better than the graded exams--I mean the old exams on the chem 32 website because, you know, I would click on ungraded exam--I mean unanswered exam and then I flip back to the answered exam--yours was just so much more convenient.

  • I downloaded some of the problems . . . oh you mean the chem website or . . . I only went to the chemweb.stanford.edu one. What is the other one?

  • Oh, you're "Kalee"? I've always been meaning to use it but I haven't gotten to it but I have a friend who has used it and said it was really good . . . (the former student's partner) I wasn't there the first day of class. I was also just using the old exams from the chemweb site . . .

  • Oh is that the one with the kaleeg . . .? I've always been meaning to go there but haven't yet. Yeah, I've used it. I thought it was pretty helpful, yes . . . but--I don't know--some of the questions were just from old tests, right? (Me: A lot of them were. Some of them were new but most of them were just old exam questions.) Some new ones would be better--we already have access to the old exams--but it does explain things a lot better than they do.

  • I think it was helpful, like, having the problems there to work out, because they were all grouped into, like, areas and topics, because the one on the Chemweb--sometimes you have downloading problems, which makes it hard. (Me: Is there anything I could add or change to make it better?) Sometimes the explanations lacked a little bit of detail, like, it just went straight to the answer instead of talking about the concepts.

  • Yeah, they've definitely been helpful. Especially in studying for the midterms. (Me: Is there anything I could add or change to make it more helpful?) Not really, because it's really nice to have the problems and solutions. Like explanations are good and stuff like that. (Me: Do you usually print stuff out or work online?) Well, when I tried to print stuff out from it--I don't know if there's something wrong with my computer or what, but I couldn't get the actual structures to print out . . . it was like I just got the text . . . so usually I just work straight from the computer.

  • I lost the url--could you write it down for me?

  • I've used the ChemWeb one. (Partner says "me too.")

  • Oh, I sent you an email! I asked a question about the six-fold symmetry of Co(en)3 before the first exam--you wrote back with the answer. (Me: Oh, right! Well, I guess you've used it then . . . do you have any suggestions for how the information could be better presented?) Oh, okay . . . yeah . . . how about also the past papers in the webpage . . . you've taken by subject and put everything by subject, but how about by exam, like the 1998 exam on the ChemWeb website. I like having both formats. It would be nice if they were on the same page; it would be much more convenient. Have you used the Math 51 page? It has a really good format . . . you can get all the answers and print out the whole page and don't have to click back and forth. (Me: Do you tend to print stuff out a lot?) Yes, a lot.

  • I thought it was really really really good and like it helped me a lot. Just 'cause, I felt like, you know I felt like I wasn't understanding the basic parts of it. (Partner hadn't heard of the site and requests the url.)

  • I don't think I've actually visited that one . . . is there a link to it from Chemweb? That's where you should put it.

  • It's a great site--yeah I loved it. It's been really helpful--it's got, like, old tests and stuff. (Partner hasn't heard of it though.) Our TA Kallie said it was a good site.

  • Oh are you kaleeg?! I'm the one who put the comment! Yeah, it's great--I like it! It helped me get an A on the chem 32 test so . . . I love it! Directly after the test, we went into NMR. It was after NMR, like coordination chem. That one's not in the same format as the other sections yet. That format's definitely good because then you can't, like, scroll down like 'okay, what do I do to solve this?' I like how you summarize in your own words (you did NMR didn't you?) . . . I also liked those additional readings--they were very helpful. (Me: Do you tend to print stuff out or work online?) I tend to do everything online. I would print out, like if there was a whole test . . . I would print it out. (person in another lab group calls across the benches: I think if I wasn't a college student who has to pay for paper I would print stuff out a lot more!!) Also, if there's a test format, like in the chemweb website there are whole tests with room to work and stuff.

  • I've looked at it just once. I went but it wasn't specifically for the test. I did the problems on the symmetry. The good examples were the ones that were, like the symmetry elements that weren't just straightforward molecules--like there was one that was a pencil, and a baseball . . .

  • I actually haven't looked at it yet (neither has partner).

  • Oh, yeah. I looked at it a while back when I was studying for exam 1.

  • I used the Chemweb one. Where is the other one? (partner=same)

  • It's good! I went on it before the midterm and did all the practice problems and it helped a lot. It really helped me! Because the answers didn't correspond to the test sometimes, you know, on the official page, so this one was better. New features . . . I like how you can go to the interactive stuff . . . I played with it for hours. They really got me off track--but that's fine, that's cool. I really can't think of anything new features . . .

  • I've never been to that site. I think I've seen it briefly, like, my friend was using it to study . . . but I've never seen it myself. Yeah, I'll probably end up using it for this midterm. (partner professes not to know it existed and asks that chemweb link to it)

  • It's your website! Oh, I use it all the time! It's so awesome! Wow. New features . . . let's see . . . I can't think of anything really, it's just really good. (Me: Do you tend to print stuff out?) Yes. I print out everything. I usually just go right to the solutions, because, like, it helps when the questions are reworded in the solutions, like I just read the question again and look at the answer. Because a lot of times I just have really no idea what to do so I just go straight to the solution.

  • I think what was helpful for me was reading the text part because the course reader pretty much doesn't explain anything, it just has like the diagrams without the explanations behind it. But when I read through the text on your site, it really helps me know what I'm doing, what I'm actually calculating . . . so that helps a lot. And the practice problems of course, the more problems the better. (Me: Do you tend to print stuff out?) Both. If it's really long I'll just print it out but for the shorter stuff I just read it online. (Me: Do you have your own printer?) Yes.

  • The Virtual Manual thing? Yeah. It's pretty good the way it is . . .

  • I like how it's organized by category. That was nice. The sample problems are really helpful right before the test. (Me: Do you tend to print stuff out?) I usually work on scratch paper and look at it on the screen. (Me: Do you have your own printer?) Yeah.

  • Yeah, I've used it. Not recently though. I used it for the first exam. You know how you have one section with all the questions, and another with all the answers? If they could be, like, more accessible at once so you can check your work as you go that would be good. Having them all together would be a bad idea because once you solve the questions you just look at the answers. (I think this student thought I was talking about the Chemweb site . . . the features he describes as frustrating were the ones I redesigned!) (Partner hasn't used it; he says he plans to for the next one.)

  • I haven't yet used it. (partner hasn't heard of it)

  • I haven't seen it. Where is it?

  • Thank you so much! The only thing I thought was first like an index of terms and their definitions and second, like, nomenclature. Because nomenclature was one thing I had to find out about for myself. This is a very useful site. (Me: Do you tend to print stuff out or read it online?) No, I don't even have a computer. I use the computers in the libraries. Like, if I want to do a [ChemWeb] test, I'll print stuff out, but to do problems, I just do them online.

B. Questionnaire Data

Two questionnaires were administered to Chem 32 students in class the mornings following the second and third exams for the course. These questionnaires tracked website usage patterns and study method time allotments for the two exams. Students were given the option of entering their names as authorization for the use of their exam scores in the data analysis.

Both questionnaires garnered twenty-nine respondents, which represents about half the class. Thirteen students authorized use of their Exam 2 scores and fifteen students authorized use of their Exam 3 scores, but only five of the authorizing names were the same between the two questionnaires. This suggests a relatively small overlap between the respondents to the first and second questionnaires. In both cases, the average score of students who volunteered their score data was a few points higher than the class average, but this difference was not significant. In cases where checkmarks instead of numbers of hours were entered, the data is represented as 'c'.

Below are reproduced the surveys as they were given to the students. Note that the two surveys are slightly different; the second survey explicitly asks whether the student is a native English speaker and also solicits specific feedback on the organization of the kinetics and thermodynamics section, which has a different format from the other sections. In order to maintain an impression of shortness (so that students wouldn't think the survey was too long or complicated to spend time on), I printed out the surveys two to a page and cut them in half.

1. November 3rd Survey and Data

I am a past Chem 32 Head-TA now in the School of Education. (I talked to many of you in lab last week.) I am trying to track usage patterns and hypertext reading strategies for the Chem 32 website that I wrote last summer (accessible from http://www.stanford.edu/~kaleeg/chem32/) and the ChemWeb one which I worked on two years ago. Whether or not you used either site, please fill out this survey and give it back to me after class. (I have braided brown hair and will be wearing a purple sweater with a dark blue jacket.) Thank you!!

1. Please fill in the approximate numbers of hours in each category that describe your studying patterns for Exam 2:

___ problems from ~kaleeg/chem32 ___ text from ~kaleeg/chem32
___ student study group ___ problems from ChemWeb
___ TA office hours ___ text from the course reader
___ other (please describe): ___________________

2. Of the following web-available study aids, please check any that you printed out in order to study for Exam 2:

___ problems from ~kaleeg/chem32 ___ solutions from ~kaleeg/chem32
___ text from ~kaleeg/chem32 ___ blank exams from ChemWeb
___ solved exams from ChemWeb

3. Do you have your own printer or do you pay for printing (in the libraries, clusters, etc.)?

4. Do you have any suggestions for how information courld be better organized on either site?

5. This is OPTIONAL!! For my education project, it would be useful to be able to correlate study patterns to grades. If you write your name here, I will use your exam 2 grade as data to study this correlation. All such data will be used ANONYMOUSLY for the purpose of recommending studying techniques to future chem 32 students. If you authorize me to use your grade data, please write your name below.

 

Hours spent by students who authorized use of their grade data (13 total):

Subject #

1

2

3

4

5

6

7

8

9

10

11

12

13

~probs

3

 

 

 

1

2

2

 

3

 

 

 

c

ChemWeb prob

3

15

5

3

1

 

 

2

0.5

1.5

2

5

c

Study Group

 

1

 

 

5

2

 

2

2

 

 

3

 

TA OH

 

 

1

2

6

2

 

 

 

2

 

3

c

~text

1

 

 

 

0.5

 

.25

 

 

 

 

 

 

Course Rdr text

1

15

10

4

1.5

4

 

3

1

1.5

5

1

c

Other

 

10*

 

 

2**

 

 

2***

 

 

 

 

 

Exam score

65.5

63

73

50.5

54.5

60

77

78.5

67.5

86.5

66.5

73

94
*Zumdahl and Atkins (textbooks)
**homework problems
***sample problems from TA's

Exam 2 class average: 64.9. Average of subjects who volunteered their grade data: 70.

What Subjects Printed Out (blue or medium gray shading means printing is NOT free; yellow or light gray shading means printing is free):

Subject #

1

2

3

4

5

6

7

8

9

10

11

12

13

~probs

 

 

 

 

y

 

 

 

 

 

 

 

 

~solns

 

 

 

 

y

 

 

 

 

 

 

 

 

ChemWeb probs

 

Y

y

 

 

Y

 

y

 

y

 

y

y

ChemWeb solns

 

Y

 

 

 

 

 

 

 

 

 

y

 

~text

 

 

 

 

y

 

 

 

 

 

 

 

 

November 3: Students who did not authorize use of their grade data:

Subject #

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

~probs

 

1

1

c

 

1

 

1

2

 

3

1

0.5

 

 

 

ChemWeb probs

 

3

3

 

3

 

4-5

3

1

2-3

0.5

1

5

4

2-3

8

Study Group

2

3

6

 

0.5

3

3-4

5

 

10

 

 

 

4

 

2

TA OH

4

1

 

 

 

 

2-3

3

 

 

 

 

1

 

5

3

~text

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0.5

Course Rdr text

2

2

3

 

3

6

4-5

3

3

2-3

 

2

10

5

1

4

Other

 

 

 

 

 

 

 

 

 

 

2*

 

 

 

 

 
* Review Session

What Subjects Printed Out (blue or medium gray shading means printing is NOT free; yellow or light gray shading means printing is free):

Subject #

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

~probs

 

 

 

y

 

 

 

 

 

 

 

 

 

 

 

 

~solns

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ChemWeb probs

 

y

Y

 

 

 

y

y

 

 

 

 

y

y

 

y

ChemWeb solns

 

y

Y

 

 

 

 

 

 

 

 

 

y

y

 

y

~text

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Suggestions (subject number in parenthesis):

  • The later problems tend to be uniformly complex and multipart. I'd appreciate simpler ones as well, to get the basics down first. Also more texts on later subjects, an index of terms, and nomenclature problems. (1)
  • ChemWeb solved exams are not always easy to see (rather illegible). Information could probably be organized better. (3)
  • Site awesome! Could you add more of the text subjects? They are really great! (7)
  • I didn't know the ~kaleeg site existed. I will check it out. (8)
  • Maybe I didn't look hard enough but I didn't see the text on ~kaleeg/ (13)
  • Sorry, haven't really been there. (14)
  • Some of the problem sets printed in symbol font (17).
  • Some links don't work -> they should be fixed (23).

November 19th Survey: The survey conducted on November 19th was similar to that conducted on November 3rd. There were two added questions: one asking whether the subject was a native English speaker and one probing responses to the re-designed text section on thermodynamics and kinetics. The key for these data is the same as that on November 3. The data follow:

Hours spent (pink/dark gray signifies a non-native English speaker):

Subject #

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

~probs

2

 

 

 

3

2

 

2

1

 

 

 

5

c

 

ChemWeb probs

 

 

3

 

4

1

 

 

3

2

4

3

 

c

 

Study Group

1

 

 

 

 

 

 

1

 

 

 

 

 

c

 

TA OH

 

 

2

2

2

4

 

 

1

 

3

2

2

c

2

~text

 

 

 

 

1

 

 

 

0.5

 

 

 

 

 

 

Course Rdr text

0.5

4

0.5

10

1

2

6

2

1

2

8

3

 

c

2

Other

 

2*

2**

3***

 

 

 

 

 

 

 

 

 

 

 

Exam score

53

40.5

74

89.5

70.5

58

65.5

71.5

83.5

86

69

70.5

69.5

48.5

67.5
*Books from Swain Chemistry Library
**Review session
***TA handouts

Exam 3 class average: 65.3.   Average score of subjects who volunteered their grade data: 67.8.

What Subjects Printed Out(blue or medium gray shading means printing is NOT free; yellow or light gray shading means printing is free):

Subject #

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

~probs

 

 

 

Y

 

 

 

 

 

 

 

 

 

 

 

~solns

 

 

 

Y

 

 

 

Y

 

 

 

 

 

 

 

ChemWeb probs

 

 

Y

Y

 

 

 

 

 

Y

Y

Y

Y

Y

 

ChemWeb solns

 

 

 

Y

 

 

 

 

 

Y

 

Y

Y

Y

 

~text

 

 

 

Y

 

 

 

 

 

 

 

 

 

 

 

Hours spent for students who did not authorize use of their grade data (14 total):

Subject #

16

17

18

19

20

21

22

23

24

25

26

27

28

29

~probs

 

3

2

1

3

 

5

3

1

 

 

 

7

 

ChemWeb probs

 

 

 

 

 

c

10

0.5

10

3

5-6

 

1.5

 

Study Group

 

1

 

 

 

c

 

 

1

1

4

2

0.5

 

TA OH

6

 

3

1

2

 

2

3.5

2

 

 

2

1

 

~text

 

 

 

 

1

 

 

 

 

3

 

 

1

 

Course Rdr text

1

 

2

5

2

c

 

1

1

 

9

4

3.5

2

Other

 

 

 

 

 

 

 

 

 

 

 

 

 

2*
*reviewed past homeworks and solutions

What Subjects Printed Out: (blue or medium gray shading means printing is NOT free; yellow or light gray shading means printing is free):

Subject #

16

17

18

19

20

21

22

23

24

25

26

27

28

29

~probs

 

 

 

 

Y

 

Y

 

 

 

 

 

Y

 

~solns

 

 

 

Y

Y

 

Y

Y

 

 

 

 

Y

 

ChemWeb probs

 

 

 

 

 

Y

 

 

Y

 

Y

 

Y

 

ChemWeb solns

 

 

 

 

 

Y

Y

 

 

 

 

 

Y

 

~text

 

 

 

 

 

 

 

 

 

vv

 

 

 

Y

 

Comments (subject number in parantheses):

  • [Kinetics/thermo review organization] extremely helpful. (5)
  • [Kinetics/thermo review organization] very helpful!! (6)
  • [Kinetics/thermo review organization] helpful. (8)
  • [Kinetics/thermo review organization] helpful. (9)
  • Re printing: I mooch always off a guy in my dorm . . . he doesn't really like Chem 32 now . . . (11)
  • I didn't know there was text on /~kaleeg/chem32. I will check it out.
  • [Kinetics/thermo review organization] helpful! Thank you! (14)
  • Re printing: ?I use my roommate's printer, and then feel guilty about it. (15)
  • [Kinetics/thermo review organization] helpful. (28)