Lab exam and peer eval info for students - #135
Open
novo52 wants to merge 24 commits into
Open
Conversation
The rubric appears both as per-objective tables on the peer evaluation page and as a one-sheet reference chart, so the descriptors live in rubric_data.py and _ext/rubric.py renders them into list-tables at build time. The chart page loads rubric_sheet.css through an html-page-context handler so the full-width, small-type layout applies to that page alone, on screen as well as in print. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxUWtBXeDptXWdHAzqdJvX
…e page overclaims
# Conflicts: # .gitignore # info/conf.py
Renders rubric_data.py as text2qti Markdown: one quiz per evaluation session, covering each evaluated member and the team. Compiles to a QTI package for import into Canvas as a graded survey. Canvas keys a replacing import on an identifier text2qti hashes from the questions alone, so the session name goes in the first question rather than only in the title. In scope: the evaluator's rubric, marks and justifications; the contribution points behind a flag. Out of scope: the pre-lab question list, and pushing marks back to Canvas.
Sir-NoChill
requested changes
Aug 21, 2026
Comment on lines
+1
to
+15
| """Directives that render the peer evaluation rubric from ``rubric_data.py``. | ||
|
|
||
| Each directive emits a reStructuredText ``list-table`` and hands it back to the | ||
| parser, so the output matches tables written by hand elsewhere in the docs. | ||
|
|
||
| - ``rubric-weights`` objective, scope and weight for all six objectives | ||
| - ``rubric-levels`` what each of the four performance levels means | ||
| - ``rubric-anchors`` the mark a given set of rubric placements is worth | ||
| - ``rubric-objective`` one objective: its lead-in and its four descriptors | ||
| - ``rubric-objectives`` every objective of a given scope, in order | ||
| - ``rubric-chart`` all objectives against all levels, as one wide grid | ||
|
|
||
| The chart page also gets its own stylesheet, attached here so that the rest of | ||
| the site keeps the theme's usual layout. | ||
| """ |
Collaborator
There was a problem hiding this comment.
Are there places in the doc where we rewrite these tables manually? Can we refactor the docs to have a single source of truth for these rubrics and point explicitly to the course info page?
Collaborator
There was a problem hiding this comment.
I had no luck with Canvas Rubriks :-( Consequently, I've been moving towards this document as the source of truth. If we want to post it (or link to it?) in other parts of the course description, what is the best format? PDF? HTML?
Contributor
Author
There was a problem hiding this comment.
@rcunrau The chart in the docs is intended to be printable to pdf or render nicely in a browser for student reference during evaluations. We can link to it.
Comment on lines
+120
to
+128
| def setup(app): | ||
| app.add_directive("rubric-weights", RubricWeights) | ||
| app.add_directive("rubric-levels", RubricLevels) | ||
| app.add_directive("rubric-anchors", RubricAnchors) | ||
| app.add_directive("rubric-objective", RubricObjective) | ||
| app.add_directive("rubric-objectives", RubricObjectives) | ||
| app.add_directive("rubric-chart", RubricChart) | ||
| app.connect("html-page-context", _attach_sheet_css) | ||
| return {"parallel_read_safe": True, "parallel_write_safe": True} |
Collaborator
There was a problem hiding this comment.
Do we have other places we can use these?
Comment on lines
-9
to
-11
| # If extensions (or modules to document with autodoc) are in another directory, | ||
| # add these directories to sys.path here. If the directory is relative to the | ||
| # documentation root, use os.path.abspath to make it absolute, like shown here. |
Collaborator
There was a problem hiding this comment.
We probably want to keep this little comment for future claude
| Lab Exams | ||
| ========= | ||
|
|
||
| Each project is followed by a lab exam: an individual, written-in-person assessment of whether you can work in a compiler codebase yourself. The exact set of exams and the weight each carries in your grade are announced with the course outline. |
Collaborator
There was a problem hiding this comment.
any way we could link to a persistent course syllabus from which we generate the version that we submit to the department? That would be ideal, then we can point to prior versions of the spec, but this is an @rcunrau call
Collaborator
There was a problem hiding this comment.
I think I currently have the assignment due dates in Canvas and on the calendar. I can add the lab exam dates to the calendar as well.
| * ``EXAM.md`` contains the exam tasks and their point values. | ||
| * ``tests/`` holds the test configuration and an empty directory for the tests you write. **No reference tests are shipped** — writing the tests that expose the behaviour you are looking for is part of the exam. | ||
|
|
||
| Read ``README.md`` first. The tasks are all stated relative to it. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Read ``README.md`` first. The tasks are all stated relative to it. | |
| Read ``README.md`` first. The tasks are all stated relative to the README |
| Submitting your work | ||
| -------------------- | ||
|
|
||
| **You are graded on what has reached GitHub by the end of the exam.** Not your working tree, not your local commits. |
Collaborator
There was a problem hiding this comment.
Suggested change
| **You are graded on what has reached GitHub by the end of the exam.** Not your working tree, not your local commits. | |
| **You are graded on what has reached GitHub by the end of the exam.** |
|
|
||
| **You are graded on what has reached GitHub by the end of the exam.** Not your working tree, not your local commits. | ||
|
|
||
| Push early and push often. A commit sitting unpushed on a lab machine when time is called is not a submission, and "it was finished locally" is not something anyone can verify afterwards. A series of pushes across the hour is also the cheapest insurance you have against the machine failing at minute fifty. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Push early and push often. A commit sitting unpushed on a lab machine when time is called is not a submission, and "it was finished locally" is not something anyone can verify afterwards. A series of pushes across the hour is also the cheapest insurance you have against the machine failing at minute fifty. | |
| Push early and push often. A commit sitting unpushed on a lab machine when time is called is not a submission. Note that **we _cannot_ verify the validity of timestamps in a local git repository as they can be spoofed**. A series of pushes across the hour is also the cheapest insurance you have against the machine failing. |
| If something goes wrong | ||
| ----------------------- | ||
|
|
||
| Machine and network failures happen. If yours fails during the exam, tell an invigilator immediately rather than trying to recover on your own — how much time you lose depends on how quickly it is reported. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Machine and network failures happen. If yours fails during the exam, tell an invigilator immediately rather than trying to recover on your own — how much time you lose depends on how quickly it is reported. | |
| Machine and network failures happen. If yours fails during the exam, tell an invigilator immediately. The teaching team can adjust your allocated time if and only if you inform them immediately. |
| * **Do your share of the project.** The exam asks for the same skills the project asks for, on a codebase you have never seen. There is no shortcut around having practised them. | ||
| * **Practise reading unfamiliar code.** Getting oriented in a codebase you did not write — finding where a construct is handled and following it through — is the first thing you do in the exam and the thing time pressure punishes most. | ||
| * **Practise debugging from a failing test.** Given wrong output, be able to work backwards to which part of the implementation produced it. | ||
| * **Know the commands.** Configuring a build, rebuilding after an edit, running ``dragon-runner``, committing and pushing — you should be typing these without stopping to think. Fumbling the build costs exam time that is not coming back. |
Collaborator
There was a problem hiding this comment.
Suggested change
| * **Know the commands.** Configuring a build, rebuilding after an edit, running ``dragon-runner``, committing and pushing — you should be typing these without stopping to think. Fumbling the build costs exam time that is not coming back. | |
| * **Know the commands.** Configuring a build, rebuilding after an edit, running ``dragon-runner``, committing and pushing. You may want to explore various text search utilities like `grep` and `rg` (ripgrep), which will be useful in searching documentation. |
|
|
||
| A vehicle is never your own submission or your teammates'. The project is graded on a working compiler, and nobody can tell from a repository alone which member of a team understood which part of it. Because the vehicle is unfamiliar, you cannot fall back on code you happen to remember writing, and a team whose work was unevenly divided does not get to hide that in the exam. | ||
|
|
||
| Being unfamiliar does not mean being undocumented. Everything you need in order to work out what a vehicle *should* do is in its repository — most importantly a spec that plays the same role as ``README.md``/``littleC_spec.md`` does for the projects: it defines correct behaviour, and it is what you check the implementation against. Read it first; every exam task is stated relative to it. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Being unfamiliar does not mean being undocumented. Everything you need in order to work out what a vehicle *should* do is in its repository — most importantly a spec that plays the same role as ``README.md``/``littleC_spec.md`` does for the projects: it defines correct behaviour, and it is what you check the implementation against. Read it first; every exam task is stated relative to it. | |
| Everything you need in order to work out what a vehicle *should* do is in its repository — most importantly a spec,``README.md`` and ``littleC_spec.md`` for respective vehicles, defines correct behaviour, and it is what you check the implementation against. Read it first; every exam task is stated relative to it. |
rcunrau
approved these changes
Aug 24, 2026
rcunrau
left a comment
rcunrau
left a comment
Collaborator
There was a problem hiding this comment.
I think this is definitely good enough to post. We can always tweak it as we get feedback through the term
**In scope** - Non-building code scores zero; local git timestamps are not evidence - Contribution points are distributed per evaluator, and the evaluated team assesses the evaluators - Vehicle specs are named SPEC.md throughout - AI assistants are prohibited whether hosted or run locally - The session monitor's behaviour is left unstated; the dry run checks for outgoing connections - A "What to bring" list, and grep/rg in the preparation advice - Rhetorical padding cut across all three pages **Out of scope** - Whether the rubric has a single source of truth across the docs, and in what format it is published - Linking the exam schedule to a persistent course syllabus
Extracts the session table and the question-writing helpers into qti_quiz.py so a second form can be generated from the same rubric source. The evaluator assessment generates byte-identically before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SSnwgk2jzZGkhTvZpTEu5h
Every member of an evaluated team gives the evaluating team one mark out of 100 for how the session was run, with a justification, reports whether they were personally given a fair chance to show their ability, and ticks off the coverage areas reached and the required questions they were asked. Serious conduct goes in a question of its own, ahead of the mark. The coverage areas and the three required questions move into rubric_data.py and are rendered onto the Peer Evaluation page, so the lists students prepare and the lists they confirm come from one source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SSnwgk2jzZGkhTvZpTEu5h
Two printable landscape pages. The cheat sheet is read from: what the evaluators have to judge, the standard the evaluated team marks the session against, follow-up question templates with blanks to fill from what the team just said, how to ask, and a clock carrying both sessions of a lab period against where each should be by a given time. The tracking sheet is written on, one per session: a row per evaluated member, a box per required question, a column for the ten contribution points, and the coverage areas with room to record who answered. Each comes out on a single sheet, checked by printing them headless. The cheat sheet's layout applies on screen as well, so a laptop shows what the printer produces. A directive that names its table sets the class on the node: rst-class resolves through a document-level transform that a parse into a detached node never reaches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SSnwgk2jzZGkhTvZpTEu5h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a page explaining the lab exams to students and a page explaining the vehicle languages each exam uses.
Adds a page explaining the peer evaluations to students.