Project 2: Program Analysis

Update April 2nd: Late submissions are allowed with penalty. You can choose to have us grade a later commit (before Sunday April 6th at 11:59pm) by adding a file "extension.txt" to your git repo that says "Please grade our Sunday April 6th submission." But, this will result in a 1 point (5%) deduction on the out-of-20 project grade.

Task Description 

• Aim: a small program analysis tool along with possible visualisation components.
• Choose a use-case / scenario related to program behaviour; e.g. one/both of:
  1. Analysing properties via a program's source code (static program analysis)
  2. Analysing properties via a program execution? (dynamic program analysis)
     These different types of analysis will be discussed in the next lectures, but are closely related to the static/dynamic checking ideas you have seen in Part I.
     Your use case should include a problem statement and intended user-group, as for Project 1.
• While the possibilities are broad and you're encouraged to be creative, we expect your projects to include at least two of the following three aspects:
  
  1. 1. The project includes static program analysis (applies to code including at least some challenging features relevant for your analysis topic; targets some property of program behaviour). If you also have a dynamic analysis part, that's no problem, of course.
     2. The project includes a non-trivial summarization over multiple inputs. This can be a method to summarize/compare results from multiple dynamic analysis runs, or a static analysis that is trying to summarize over all executions
     3. The project includes a substantial visualisation component, summarising and presenting your analysis data in a useful / appealing way.
• You can use frameworks / existing code (attributing sources) for this project, but some non-trivial implementation of your own must be planned - check plans with your TA (e.g. you can’t just plug an existing program analyser into a compatible visualisation tool!)
• Sketch your planned analysis informally on paper: what will its input/output be? What will the main stages be? What data needs to be passed between stages?
• Divide the project/implementation work into clearly-defined responsibilities
  
  • Make sure to use version control throughout your project; use the git repository we will provide to you.
  • Define which parts of the work you plan to be completed by which check-ins!
• Perform a task-driven prototype study on the usefulness of your idea before implementing! If you prefer to perform another kind of user study to gather early feedback, this is OK (but check your plan with your TA first).
  • Reflect on results; modify the proposal based on the results of the study; redesign accordingly.
• Implement the analysis (+ visualisation) design you've proposed.
• Perform a task-driven user study on the effectiveness of the completed product: reflect on the outcome -  what is your evaluation? If you prefer to perform another kind of user study to gather early feedback, this is OK (but check your plan with your TA first).
• Make a 3 minute video, capturing your entire process, user studies and their outcome, and evaluating the design and usefulness of your analysis (and visualisation). Make sure that you make clear which type(s) of program analysis your project includes, and why you made this choice. You should submit your video on your version control (either as a file upload or a link to video stored somewhere else). Make sure to stick to 3 minutes!

We'll likely follow up with some lightweight testing of our own; we might ask you to briefly meet with your TA mentor to assist with this testing.

Some points on grading (cf. Part II Learning Objectives: Slide Deck 7, Slide 3)

We'll use criteria related to the skills you're expected to apply in the second half of the course, according to the Learning Objectives (LOs). In particular, we'll consider:

• The original motivation for your analysis; there should be a clear and convincing use-case (considering the users it's aimed at) (LO XI).
• The design of your program analysis; to what extent does this seem a good fit for the use-case (task and users)?  (LO IIX). If applicable (usually for a static analysis), a sensible choice of trade-offs w.r.t. approximating information about possible executions (LO IX).
• Clear statements about which kind(s) of program analysis (static vs. dynamic vs. anything else) your project incorporates; a brief justification for why you made that choice (LO VII).
• The quality of your implementation in terms of a smooth user experience (including how easy it is to run and how information is presented). Including reasonable documentation outlining how to run your tool, as well as the known limitations, will help in grading. (LO X). While you're encouraged to follow good software engineering practice (clear code, organised, comments, etc.) we won't perform code review as a rule, unless wanting to check something specific.
• User evaluation during the project: performing user studies and acting appropriately on your feedback. Summarize the contents of the user studies and the feedback you got somewhere in the final repo (e.g., Checkins.md or Userstudies.md) (LO XII).
• Your own evaluation of your work at various stages, including analysis of the user study results. Identifying some negative points in your design/implementation won't necessarily reflect badly on your grade; instead, if you explain your judgement clearly with respect to the principles taught in the course then this can be a positive point (LOs XI and XII).
• Team management and organisation: dividing up and coordinating responsibilities within your group, submission of the check-ins on time, communication with your TAs, for the final submission: the video sticking to the right length (without alarming amounts of speeding up etc.). We will also consider in the grade whether at least some group member was present at the video fest to answer questions. 

Check-ins (note: this information is analogous to that for Project 1, except for some additional planning details due in Check-in 2)

Project check-ins are (lightweight) subgoals during the project, used to document your process and get explicit feedback from your TA (of course, you're also welcome to talk with your TA about other things!). Each check-in requires you to communicate information with your TA, get their feedback and add a short corresponding description to your project repositories. By default, you should speak with your TA during their office hours; it might take 10-15 minutes to talk through your check-in. Alternatively, if you arrange this in advance with your TA, you could either Zoom with them separately or communicate via email; it's your team's responsibility to organise how you'll get your check-ins signed off if you don't attend the office hours. Once you've had TA feedback on the check-in, commit an entry to a CHECKINS.md file in your project git repository including a summary of the relevant information and any relevant notes from your discussion with your TA (this can be a paragraph or two; longer if you would like). The entry should at least briefly summarise the points from the check-in. Make sure to push your commit by the check-in deadline!

Check-in deadlines are by Friday 5pm for weeks 8, 9, 10, 11 and 12 of the course (by the end of week 13 you'll submit the finished project!). Note that nothing stops you from discussing and submitting these check-ins early if you'd like to (you could even submit more than one at once if you think you're ready). You can of course also do the corresponding work on your projects at any time; the check-ins are the latest you should have thought through these points, but not necessarily the recommended times to do these parts of your project work.

Check-in 1 (Week 8): Brief description of your discussions within your team so far, and any current candidate ideas for your project. You should talk with your TA/Caroline as soon as possible about these ideas; due to the project start mid-week it's OK if you have not yet done this, but make sure to note the progress you have made so far. Any planned follow-up tasks for the next week.

Check-in 2 (Week 9): Proposal of your planned program analysis (and visualisation, if applicable), to be uploaded to GitHub. Notes of any important changes/feedback from TA discussion. Any planned follow-up tasks or features still to design. Planned division of main responsibilities between team members. Roadmap for what should be done when, including specific goals for completion by future check-ins. Summary of progress so far. Individual surveys about team collaborations also required; details in separate Assignment description.

Check-in 3 (Week 10): Mockup of how your project is planned to operate (as used for your first user study). Include any sketches/examples/scenarios. Notes about first user study results. Any changes to original design.

Check-in 4 (Week 11): Status of implementation so far. Plans for final user study. Planned timeline for the remaining days. Individual surveys about team collaborations also required; details in separate Assignment description.

Check-in 5 (Week 12): Status of final user study; any feedback and changes planned. Plans for final video (possible draft version). Planned timeline for the remaining days.

I hope you enjoy the projects! As usual, if you have any questions, please ask in Piazza or pop along to an office hour.

Best wishes,

Caroline and the TAs