Cornell University
School of Electrical and Computer Engineering
ECE 2400 / ENGRD 2140
Computer Systems Programming
Fall 2020
Prof. Christopher Batten
185 Statler Hall • Monday, Wednesday, Friday • 3:00–3:50pm
home | syllabus | schedule | handouts | resources | sponsors
Course Information
Cross-Listings | ENGRD 2140 Computer Systems Programming |
---|---|
Instructor | Prof. Christopher Batten, 323 Rhodes Hall Office Hours: Virtual, Thursday, 9:00–10:00am and 4:30–6:00pm |
Graduate TAs |
Peitian Pan, pp482, Office/Lab Hours: Thu, 7:30–9:30pm Kexin (Grace) Zheng, kz73, Office/Lab Hours: Wed, 7:30–9:30pm |
Undergraduate TAs |
Chris O'Brian, co253, Office/Lab Hours: Thu, 7:30–9:30pm Eric Hall, ewh73, Office/Lab Hours: Tue, 7:30–9:30pm Raghav Kumar, rk524, Office/Lab Hours: Wed, 7:30–9:30pm Sonal Parab, ssp92, Office/Lab Hours: Wed, 7:30–9:30pm Krithik Ranjan, kr397, Office/Lab Hours: Mon, 8:00–10:00am Janice Wei, cw655, Office/Lab Hours: Tue, 7:30–9:30pm |
Lectures | 185 Statler Hall, Mon/Wed/Fri, 3:00–3:50pm |
Disc. Section | 225 Upson Hall, Monday, 5:20–6:10pm, 7:30–8:20pm, & 8:40–9:30pm |
Texbook |
zyBook: ECE 2400 Computer Systems Programming Link: http://learn.zybooks.com Code: CORNELLECE2400BattenFall2020 Cost: $88 |
Staff Email | ece2400-staff-l cornell edu |
Objectives
This course is meant to be a foundational course in computer systems programming. The course will prepare students for more advanced coursework in computer engineering (e.g., embedded systems, computer architecture) as well as more advanced coursework that focuses on a single type of computer systems software (e.g., compilers, operating systems, databases). By the end of this course, students should be able to:
- describe a variety of algorithms and data structures and how to analyze these algorithms and data structures in terms of time and space complexity;
- apply the C and C++ programming languages to implement algorithms and data structures using a variety of programming paradigms;
- evaluate various algorithm and data structure alternatives and make a compelling theoretical and/or practical argument for why one approach is superior;
- create non-trivial C/C++ programs (roughly 1,000 lines of code) and the associated testing strategy starting from an English language specification; and
- write concise yet comprehensive technical reports that describe a program implemented in C/C++, explain the testing strategy used to verify functionality, and evaluate the program to characterize its performance and memory usage.
Prerequisites
This course is targeted towards sophomore-level undergraduate students, although it is also appropriate for advanced freshman students and upperclassman. An introductory course on computing is required. Students need to be comfortable using at least one programming language (e.g., Python through CS 1110 or MATLAB through CS 1112) and should have some experience in software design, development, and testing. No prior knowledge of the C or C++ programming languages is necessary.
Topics
The course includes four parts. The first two parts covers procedural programming in C and then uses C to explore basic algorithms and data structures, while the second two parts cover multi-paradigm programming in C++ and then uses C++ to explore more algorithms and data structures. A tentative list of topics for each part is included below. The exact topics covered in the course are subject to change based on student progress and interest.
-
Part 1: Procedural Programming
- Topic 1: Introduction to C (functions, conditional & iteration statements)
- Topic 2: C Recursion (single vs. multiple recursion)
- Topic 3: C Types (builtin, user-defined, type checking/inference/conversion/casting)
- Topic 4: C Pointers (call-by-value/by-pointer, conceptual storage vs machine memory)
- Topic 5: C Arrays (iterating, arrays as function parameters, strings)
- Topic 6: C Dynamic Allocation (malloc/free, heap)
-
Part 2: Basic Algorithms and Data Structures
- Topic 7: Lists and Vectors (interface, impl, singly/doubly linked list, bounded/resizable vec)
- Topic 8: Complexity Analysis (time and space complexity)
- Topic 9: Sorting Algorithms (insertion, selection, merge, quick, hybrid)
- Topic 10: Stacks, Queues, Sets, Maps
- Topic 11: Standard C Libraries (stdc, glib)
-
Part 3: Multi-Paradigm Programming
- Topic 12: Transition to C++ (namespaces, functions, references, exceptions, new/delete)
- Topic 13: Object-Oriented Programming (C++ classes & inheritance, dynamic polymorphism)
- Topic 14: Generic Programming (C++ templates, static polymorphism)
- Topic 15: Functional Programming (C++ functors and lambdas)
- Topic 16: Concurrent Programming (C++ threads and atomics)
-
Part 4: More Algorithms and Data Structures
- Topic 17: Trees (binary search trees, heaps)
- Topic 18: Tables (lookup tables, hash tables)
- Topic 19: Graphs (DFS, BFS, shortest path first, minimum spanning trees)
- Topic 20: Standard C++ Libraries (stl, boost)
Required Materials
The required textbook for the course is the following custom zyBook assembled specifically for this course by the instructor:
- Title: ECE 2400 Computer Systems Programming
- Link: http://learn.zybooks.com
- Code: CORNELLECE2400BattenFall2020
- Cost: $88
zyBooks are online, interactive textbooks that makes it easy and engaging to learn principles and then immediately put them out in practice using a series of integrated participation and challenge activities. The zyBook for this course includes a web-based C/C++ development tool that makes it trivial to try out small code snippets. The zyBook includes one chapter for each of the 20 topics covered in the course.
The assigned readings will be available on the Canvas course site. Students are expected to complete approximately 50% of the participation and challenge activities, and their overall progress on these activities will factor into the participation portion of their final grade. In addition, selected challenge activities will serve as homework and are due on Mondays before lecture.
The first chapter of the zyBook is available free of charge to students after they create a zyBook account. Students can complete zyBook activities in the first chapter while they are finalizing their decision to enroll in the course. Although their work will be saved, this work will not be visible to the instructors until students actually purchase the zyBook. Note that zyBook will offer a full refund for students that drop the course within the first few weeks of the semester. Email support@zybooks.com to request a refund.
Format and Procedures
This course includes a combination of lectures, short in-class quizzes, optional discussion sections, assigned readings, programming assignments, and exams.
- Lectures – Lectures will be from 3:00pm to 3:50pm every Monday, Wednesday, and Friday in 185 Statler Hall through November 13, excluding the break on October 14. We will start promptly at 3:00pm so please arrive on time. All lectures will also be available live via Zoom, and also recorded via Zoom. After the Thanksgiving break, all lectures will only be via Zoom. Students are expected to make every effort to attend all lectures (either in-person or remote), be attentive during lecture, and participate in class discussion. Although class participation will be assessed as part of each student's final grade, we also understand that some remote access students may be in timezones which prevent attending the lectures synchronously. These students should reach out to the instructor early in the semester to seek an accommodation. Please turn off all cellular phones during class. Use of cellular phones and laptops during lecture is not allowed for in-person students (see policy section).
- Discussion Section – The discussion sections will be on Mondays at 5:20–6:10pm, 7:30–8:20pm, or 8:40–9:30pm in 225 Upson Hall. All discussion sections will also be available live via Zoom. They may be recorded, and the corresponding handouts will be available for students to complete on their own time. Students are expected to make every effort to attend the weekly discussion sections. Although discussion section participation will be assessed as part of each student's final grade, we also understand that some remote access students may be in timezones which prevent attending the lectures synchronously. These students should reach out to the instructor early in the semester to seek an accommodation. These discussion sections will be relatively informal, with the primary focus being on facilitating student's ability to complete the programming assignments and on reviewing material from lecture using problem-based learning.
- Readings & Homework – Students are expected to complete all of the required reading according to the schedule on the course website. There will be assigned readings in the zyBook, and students will need to complete the homework corresponding to each of these readings. The homework will be in the form of selected challenge activities in the zyBook. Homework is due on Mondays at 3:00pm (before lecture).This will enable students to either prepare for a future lecture or review material recently presented in lecture.
- Online Discussion – Engagement via the Piazza online discussion forums is a critical component of the course. Students are required to actively participate on Piazza, and this participation will be assessed as part of each student's final grade. There are many ways student can participate including posting questions, liking other questions, responding that they have the same question, helping to create answers, and posting interesting related content.
- Programming Assignments – The course will include five programming assignments. Students are expected to work individually on the first three programming assignments and in a group on the final two programming assignments. Students will be using the ECE Computing Resources to complete the programming assignments, the code must be submitted via GitHub, and the report must be submitted in PDF format via the online Canvas assignment submission system (see resources section). Students are also required to write a short two- to three-page document describing the test strategy used in the first three programming assignments. No other means of submission will be accepted. Programming assignments are due on Thursdays at 11:59pm except for the final programming assignment which will be due during the final exam period (see policy section for late assignment policy).
- Exams and Quizzes – The course includes one prelim exam, one semi-final exam, and one short take-home quiz. The exams and quizzes assess student understanding of the material presented in lecture and assigned readings. The prelim exams will include a series of short answer questions and cover the first two parts of the course. The semi-final exam will cover the first three parts of the course. The semi-final exam will include two sections: the first section will include a series of short answer questions, while the second section will include a detailed design problem which will require students to compare and contrast two design alternatives. There will also be a short take-home quiz due at the end of the semester which will cover the fourth part of the course. The exams and quizzes do not assess student understanding of the Linux development environment (e.g., the Linux command line, make, gdb, etc.) used for the programming assignments. If students have a scheduling conflict with the prelim or semi-final exam, they must let the instructor know as soon as possible, but no later than two weeks before the prelim or semi-final exam. Graded exams and exam solutions are only available for review under the supervision of a course instructor. You may not remove your graded exam, nor may you remove the exam solutions.
Grading Scheme
Each part or criteria of every assignment is graded on a four-point scale. A score of 4.25 is an A+, 4 roughly corresponds to an A, 3 roughly corresponds to a B, 2 roughly corresponds to a C, and below a 2 roughly corresponds to C- or lower. A score of 4.0 usually indicates that the submitted work demonstrates no misunderstanding (there may be small mistakes, but these mistakes do not indicate a misunderstanding) or there may be a very small misunderstanding that is vastly outweighed by the demonstrated understanding. A score of 3.0 usually indicates that the submitted work demonstrates more understanding than misunderstanding. A score of 2.0 usually indicates that the submitted work demonstrates more misunderstanding than understanding. A score of 1.0 usually indicates that the submitted work is significantly lacking in some way. A score of 4.25 is reserved for when the submitted work is perfect with absolutely no mistakes or is exceptional in some other way.
Total scores are a weighted average of the scores for each part or criteria. Parts or criteria are usually structured to assess a student's understanding according to four kinds of knowledge: basic recall of previously seen concepts, applying concepts in new situations, qualitatively and quantitatively evaluating alternatives, and creatively implementing new designs; these are ordered in increasing sophistication and thus increasing weight. In almost all cases, scores are awarded for demonstrating understanding and not for effort. Detailed rubrics for all quizzes, programming assignments, and exams are provided once the assignment has been graded to enable students to easily see how the score was awarded. A detailed Programming Assignment Assessment Rubric is available on the public course webpage.
The final grade is calculated using a weighted average of all assignments. Participation will be assessed in part based on: (1) zyBook participation activities; (2) lecture attendance and participation; (3) discussion section attendance and participation; (4) online Piazza discussion; and (5) completing student evaluations. The weighting for the various assignments is shown below.
Participation | 5% |
Homework | 10% (assigned zyBook challenge activities) |
PA Milestones | 5% (weighted equally) |
PA Code | 25% (weighted equally) |
PA Reports | 10% (weighted equally, includes testing doc) |
Prelim Exam | 15% |
Semi-Final Exam | 25% |
Short Take-Home Quiz | 5% |
Note that the exams account for roughly 40% of a student's final grade. The exams in this course are very challenging. Successful students begin preparing for the exams far in advance by carefully reviewing the assigned readings, independently developing study problems, and participating in critical study groups.
To pass the course, a student must at a bare minimum satisfy the following requirements: (1) submit four out of the five programming assignments; (2) take the prelim exam; (3) take the semi-final exam; and (4) submit the short take-home quiz. If a student does not satisfy these criteria then that student may fail the course regardless of the student's numerical grade. The instructor reserves the right to award a D letter grade for students who barely satisfy this criteria but are clearly making no real effort to engage in the course and their own learning.
Degree Requirements
This section describes how ECE 2400 can satisfy various degree requirements. Students are responsible for confirming this information with the appropriate student services coordinator.
- ECE Undergraduate Majors – We strongly encourage all ECE undergraduate majors to consider taking ECE 2400. ECE 2400 / ENGRD 2140 can be used as your second engineering distribution course (all ECE students must take ECE 2300 as their first engineering distribution course). Alternatively, ECE students may be able to use this course as an outside-ECE technical elective. Students are reminded they are only allowed to use one ECE course as an outside technical elective provided that the course's subject matter lies outside the student's major disciplinary area as determined by the focus of the student's upper-level ECE course work. Regardless, this course satisfies the ECE advanced programming requirement.
- ECE Undergraduate Minors – ECE 2400 / ENGRD 2140 can potentially be used as an engineering distribution course (see "Other Undergraduate Students" below) towards your major requirements. However, ECE 2400 does not satisfy any of the specific requirements for the ECE undergraduate minor.
- CS Undergraduate Majors – CS majors should take CS 2110 / ENGRD 2110 as their first engineering distribution course. Since both CS 2110 / ENGRD 2110 and ECE 2400 / ENGRD 2140 are part of the same "scientific computing" ENGRD category, you cannot use ECE 2400 as your second engineering distribution course. However, CS majors can still use ECE 2400 as an advisor-approved elective or a major-approved free elective.
- ECE/CS Undergraduate Double Majors – A recent change to the CS course requirement means that ECE/CS double majors can now take ECE 2400 / ENGRD 2140 instead of CS 2110 / ENGRD 2110. Taking ECE 2400 no longer complicates the path to a double major.
- CS Undergraduate Minors – ECE 2400 / ENGRD 2140 can now also be substituted for CS 2110 to satisfy the specific requirements for the CS undergraduate minor. We encourage CS undergraduate minors wishing to focus on computer systems to consider taking ECE 2400 instead of CS 2110.
- Other Undergraduate Students – Students are required to take two ENGRD courses from different categories. ECE 2400 / ENGRD 2140 is part of the "scientific computing" category, so students can use this course as their second engineering distribution course assuming the first is not also in the "scientific computing" category (i.e., not ENGRD 2110, ENGRD 2112, ENGRD 3200). ECE 2400 / ENGRD 2140 is a great way to strengthen your programming skills and can nicely complement core classes in many other disciplines.
- ECE M.Eng. Students – ECE 2400 can be used as a technical elective course, although students are reminded they can only use one ECE undergraduate course as a technical elective.
Policies
This section outlines various policies concerning auditors, usage of cellular phones and laptops in lecture, turning in assignments late, regrading assignments, collaboration, copyright, and accommodations for students with disabilities.
Auditor Policy
Casual listeners that attend lecture but do not enroll as auditors are not allowed; you must enroll officially as an auditor. If you would like to audit the course please talk to the instructor first! Usually we wait until the second week of classes before allowing auditors to enroll, to ensure there is sufficient capacity in the lecture room. The requirements for auditors are: (1) attend most of the lectures; (2) complete a reasonable amount of the zyBook participation activities; and (3) complete a reasonable amount of the zyBook challenge activities. This does mean you have to buy the zyBook. If you do not plan on attending the lectures for the entire semester or you do not plan on doing any of the reading, then please do not audit the course. Please note that students are not allowed to audit the course and then take it for credit in a later year unless there is some kind of truly exceptional circumstance.
Course Re-Enrollment Policy
Students are not allowed to enroll for credit for a significant fraction of the course, drop or switch to auditor status, and then re-enroll for credit in a later year. A "significant fraction of the course" means after the first prelim; by this time the student will have: attended several lectures, completed multiple programming assignments, and completed several short quizzes. The student should have plenty of experience to decide whether or not they should drop and take the course in a later year. It is not fair for students to have access to assignment solutions and possibly even take both prelims before deciding to drop the course and take it again in a later year; this would essentially enable students to take the course twice to improve their grade.
Cellular Phones and Laptops in Lecture Policy
Students are prohibited from using cellular phones and laptops in lecture unless they receive explicit permission from the instructor. It is not practical to take notes with a laptop for this course. Students will need to write on the handouts, quickly draw state diagrams, and sketch pseudocode during lecture. The distraction caused by a few students using (or misusing) laptops during lecture far outweighs any benefit. Tablets are allowed as long as they are kept flat and used exclusively for note taking. If you feel that you have a strong case for using a laptop during lecture then please speak with the instructor.
Late Assignment Policy
Programming assignment reports must be submitted electronically in PDF format and the code must be submitted electronically via GitHub. No other formats will be accepted! Programming assignments must be submitted by 11:59pm on the due date. No late submissions will be accepted and no extensions will be granted except for a family or medical emergency. We will be using the online Canvas assignment submission system. You can continue to resubmit your files as many times as you would like up until the deadline, so please feel free to upload early and often. If you submit an assignment even one minute past the deadline, then the assignment will be marked as late.
As an exception to this rule, each student has five slip-days that may be used when submitting programming assignments (both incremental milestones and the final assignment) throughout the semester. Each slip-day provides an automatic 24-hour extension. The PA code and report are treated as a single assignment for the purposes of slip days (i.e., a single slip day provides an automatic 24-hour extension for submitting a student's code, report, or both). When working with a partner, both students must have a slip day to secure an automatic 24-hour extension. Regardless, the maximum automatic extension is 48 hours. The purpose of the slip-day system is to give you the freedom to more effectively manage your time. The due dates for the course are available at the beginning of the semester, so please plan ahead so you can handle weeks with many other deadlines. To use a slip day, simply submit the report late; Canvas will allow assignments to be uploaded up to two days late. You are responsible for keeping track of how many slip days you have remaining. If you accidentally submit an assignment late without the proper number of slip days remaining then, although the system will allow the upload, we will not grade the assignment (or we will grade the latest upload before the due date).
Regrade Policy
Addition errors in the total score are always applicable for regrades. Regrades concerning the actual solution should be rare and are only permitted when there is a significant error. Please only make regrade requests when the case is strong and a significant number of points are at stake. Regrade requests should be submitted online via a private post on Piazza within one week of when an assignment is returned to the student. You must provide a justification for the regrade request.
Collaboration Policy
The work you submit in this course is expected to be the result of your individual effort only, or in the case of the final two programming assignments, the result of you and your group's effort only. Your work should accurately demonstrate your understanding of the material. The use of a computer in no way modifies the standards of academic integrity expected under the University Code.
You are encouraged to study together and to discuss information and concepts covered in lecture with other students. You can give "consulting" help to or receive "consulting" help from other students. Students can also freely discuss basic computing skills or the course infrastructure. However, this permissible cooperation should never involve one student (or group) having possession of or observing in detail a copy of all or part of work done by someone else, in the form of an email, an email attachment file, a flash drive, a hard copy, or on a computer screen. Students are not allowed to seek consulting help from online forums outside of Cornell University. Students are not allowed to use online solutions (e.g., from Course Hero) from previous offerings of this course. Students are encouraged to seek consulting help from their peers and from the course staff via office hours and the online Piazza discussion forums. If a student receives consulting help from anyone outside of the course staff, then the student must acknowledge this help on the submitted assignment.
During in-class quizzes and examinations, you must do your own work. Talking or discussion is not permitted during the examinations, nor may you compare papers, copy from others, or collaborate in any way. Students must not discuss a quiz/exam's contents with other students who have not taken the quiz/exam. If prior to taking it, you are inadvertently exposed to material in an quiz/exam (by whatever means) you must immediately inform an instructor.
Should a violation of the code of academic integrity occur, then a primary hearing will be held. See https://theuniversityfaculty.cornell.edu/academic-integrity for more information about academic integrity proceedings.
Examples of acceptable collaboration:
- Ben is struggling to complete a programming assignment which requires implementing a linked list. He talks with Alice and Cathy and learns that all three students are really struggling. So the three students get together for a brainstorming session. They review the lecture and reading materials and then sketch on a whiteboard some ideas on how to implement a linked list. They might also sketch out some code snippets to try and understand the best way to implement the data structure. Then each student independently writes the code for the assignment and includes an acknowledgment of the help they received from the other students. At no time do the students actually share code.
- Alice and Amy are having difficulty figuring out difficult test cases for their handwriting recognition system. They post on Piazza to see if anyone has some general ideas for tricky corner cases. Ben and Bob figured out an interesting test case that ensures their handwriting recognition system correctly handles the worst case input, so Ben and Bob post a qualitative description of this test case. Alice and Amy independently write the code for this test case and then include an acknowledgment of the help they received from the other group. At no time do the groups actually share test code.
Examples of unacceptable collaboration:
- Ben is struggling to complete a programming assignment which requires implementing a linked list. He talks with Alice and Cathy and learns that all three students are really struggling. So the three students get together for a joint coding session. Each student works on one method of the linked list class, and then they combine these methods together to create the final working linked list class. The three students share and copy each other's code often in order to finish the assignment. Each student submits the final code independently. Each student acknowledges the help he or she received from the other students, but it doesn't matter since they explicitly shared code.
- Alice and Amy are having difficulty figuring out difficult test cases for their handwriting recognition system. They post on Piazza to see if anyone has some general ideas for tricky corner cases. Ben and Bob figured out an interesting test case that ensures their handwriting recognition system correctly handles the worst case input, so Alice and Amy send their test code to Ben and Bob via email. Alice and Amy modify this test code and then include it in their submission. Alice and Amy include an acknowledgment of the help they received from the other group, but it doesn't matter since they explicitly shared code.
Notice that the key is that students should not share the actual solutions or code with each other. Consulting with your fellow students is fine and is an important part of succeeding in this course.
Copyright Policy
All course materials produced by the course instructor (including all handouts, tutorials, homeworks, quizzes, exams, videos, scripts, and code) are copyright of the course instructor unless otherwise noted. Download and use of these materials are permitted for individual educational non-commercial purposes only. Redistribution either in part or in whole via both commercial (e.g., Course Hero) or non-commercial (e.g., public website) means requires written permission of the copyright holder.
Accommodations for Students with Disabilities
In compliance with the Cornell University policy and equal access laws, the instructor is available to discuss appropriate academic accommodations that may be required for students with disabilities. Requests for academic accommodations are to be made during the first three weeks of the semester, except for unusual circumstances, so arrangements can be made. Students are encouraged to register with Student Disability Services to verify their eligibility for appropriate accommodations.
Online and Computing Resources
We will be making use of a variety of online websites and computing resources.
- Public Course Website –
http://www.csl.cornell.edu/courses/ece2400
This is the main public course website which has the course details, updated schedule, reading assignments, and most handouts. We intend for all course content to always be available on Canvas. The public course website is just for public access to some of this content. - Canvas –
http://www.csl.cornell.edu/courses/ece2400/canvas
We will be using Canvas for the first time this year to manage announcements, course content, assignment submission, and grade distribution. - Piazza Discussion Forums –
http://www.csl.cornell.edu/courses/ece2400/piazza
Piazza is an online question-and-answer platform. We will be using Piazza for all announcements and discussion on course content and programming assignments. We will enroll students that sign up for the course in Piazza. The course staff is notified whenever anyone posts on the forum and will respond quickly. Using the forum allows other students to contribute to the discussion and to see the answers. Use common sense when posting questions such that you do not reveal solutions. Please prefer posting to Piazza as opposed to directly emailing the course staff unless you need to discuss a personal issue. Please post non-anonymously if at all possible. - ECE Computing Resources
– ecelinux.ece.cornell.edu
The ECE department has a cluster of Linux-based workstations and servers which we will be using for the programming assignments. You can access the ECE computing resources remotely using x2go. More information will be made available to students shortly. - GitHub –
http://www.csl.cornell.edu/courses/ece2400/github
GitHub is an online Git repository hosting service. We will be using the commerical GitHub service to distribute programming assignment harnesses and as a mechanism for student collaboration on the final two programming assignments. Students will also use GitHub for submitting the code for their programming assignments. Students are expected to become familiar with the Git version control system. Note that we are not using the Cornell hosted version of GitHub as in some other courses; we are using github.com. - TravisCI –
http://www.csl.cornell.edu/courses/ece2400/travisci
TravisCI is an online continuous integration service that is tightly coupled to GitHub. TravisCI will automatically run all tests for a students' programming assignment every time the students push their code to GitHub. We will be using the results reported by TravisCI to evaluate the code functionality of the programming assignments. - Codecov.io –
http://www.csl.cornell.edu/courses/ece2400/codecovio
Codevoc.io is an online code-coverage visualization service that is tightly coupled to GitHub and TravisCI. Codecov.io will automatically display code-coverage reports after each TravisCI build. We will be using the results reported by Codecov.io to help evaluate the verification quality of the programming assignments.