Common Evaluation Criteria
Jurors are asked to evaluate your submission using four criteria: concept, novelty, interest, and quality. The final submission score is based on a combination of these factors. For example, a high-quality Talk that has broad appeal and is unlike other recent SIGGRAPH Talks has a good chance of acceptance, while a poorly motivated submission of interest to few attendees (or that duplicates recent Talks) probably will be rejected. Work is evaluated on the quality and completeness of the materials submitted.
Examples of accepted content from past conferences, including Talks, have been made freely available by the ACM SIGGRAPH organization and can be accessed here.
Concept
How exceptional are the ideas, problems, solutions, aesthetics, etc., presented in this submission? How coherently does the submission convey its overall concept? Is the concept similar to existing ones, or does it stand out? This criterion is particularly applicable to submissions that pull together existing technologies into a single product (for example, demos, animations, art pieces). Submissions of this type, where the individual technologies are not necessarily new but their combination is, are evaluated on both the final product and how well proposed technologies integrate to meet the desired goals. Many submissions in this area are rejected because they do what existing systems do, and they do not demonstrate that the proposed approach leads to better results.
Novelty
How new and fresh is this work? Is it a new, groundbreaking approach to an old problem, or is it an existing approach with a slightly new twist? You must first demonstrate to the jury that your work is sufficiently different from existing approaches. Second, you should evaluate your work in the context of other approaches where appropriate. Is it faster? Easier to use? Does it give better results? Is it more accurate? Many submissions are rejected either because the work is too similar to existing work or because the submission materials did not convince the jury that the improvements were substantial enough.
Interest
Will conference attendees want to see this? Will it inspire them? Are the results or approach appealing to a broad audience? This is partly a measure of how broad the potential audience is and partly a measure of the overall clarity and novelty of the submission. A submission in a very niche area is more likely to be accepted if the results are exceptionally better than what exists already or if the proposed solution might be applicable to other areas.
Quality, Craft, and Completeness
This is a measure of how well written the abstract is and the quality of the supporting materials. The abstract must effectively communicate both the problem and the solution in enough detail and clarity that the jury can evaluate it. You must also convince the jury that your solution works. Many submissions are rejected because, although the problem and solution seemed interesting, the materials did not convince the jury that the solution had actually been implemented and evaluated. If your submission has an animation, simulation, or interactive component, then including a video is essential.
Although submitters simply submit a Talk, during the review process, the jury may sort Talk submissions into focus categories for some long-established communities within SIGGRAPH: research, production, and education. This sorting is used in scheduling accepted Talks into meaningful sessions and to provide “at-a-glance” information to attendees. The primary reason for rejection of each type of Talk is given below, to help submitters understand what the jury will be looking for in typical Talk submissions.
Research Talks
Accepted Research Talks typically fall into one of two categories: an exploration of a new problem or a novel approach to an existing problem. For Talks on new problems, the jury accepts those they believe will interest attendees and inspire subsequent discussions or research. For Talks on solving existing problems, the jury accepts those that clearly solve a problem of interest to many attendees. Primary reasons a Talk is rejected include:
- The jury was unconvinced the work solves a new or existing problem. This can happen either because the abstract did not clearly differentiate the work from existing work, or the proposed solution was too incremental.
- The jury was unconvinced that there was sufficient improvement over existing work. It is not sufficient for the approach to simply be new; the submission materials also must demonstrate that the proposed approach works better (it is faster, more accurate, uses less memory, easier to use, etc.) than existing work.
- The submission materials did not clearly convey both the problem and the proposed solution. If the jury has to struggle to understand the submission, they are unlikely to accept it. Good abstracts first provide a concise statement of the problem and solution, and then provide sufficient detail to convince the jury the submitter would present a compelling Talk.
- The area is off topic and unlikely to be of interest to SIGGRAPH attendees.
- The jury believes the Talk will be an extended advertisement for a product.
Production Talks
Accepted Production Talks typically explore solutions to problems frequently encountered in production environments. They should be motivated by unique visual results or the production pipeline rather than production scope, size, or budget. Examples include new applications of research ideas in a production setting, combining existing techniques in new and unique ways, or improvements to pipeline tools or workflow for improved efficiency. The jury accepts Production Talks that will interest attendees seeking details on production difficulties and their solutions, or because the technical details may interest the broader SIGGRAPH community. Production Talks are not limited to film and visual effects but could cover other application and production environments such as game development, mobile graphics, applications of interactive techniques, immersive environments, and art and design installations.
The abstract needs to provide context for the work and aesthetic goals, underlying technical solution, and some kind of evaluation metrics. To support the submission’s claims, we strongly encourage the inclusion of some kind of visual or video material, either work in progress or the finished result. The jury frequently rejects unsubstantiated submissions. It is possible to submit material for viewing only at the jury meeting (when it has not been approved for public display or release). Please contact the Talks Chair to make arrangements.
Primary reasons that a Production Talk is rejected:
- The jury was unconvinced the submission provides a substantially new solution to a production problem. Exceptions may be made when the solution is only known to a small community; in this case, clearly acknowledge previous work and explain how this Talk reaches a broader audience.
- Workflow improvements are not supported by an objective measure (for example, a reduction in render time or short turnaround).
- Critical visual media are missing, making it difficult for the jury to judge the approach in practice. If media will be viewable only at the jury meeting, make that clear in the submission so jurors understand the images are only stand-ins.
- The Talk fails to cite existing work or explain differences from existing approaches. Although Talks need not be as rigorous as research papers, a clear discussion of the historical context is important.
- For large productions with multiple Talk submissions, the jury may feel that there is some overlap, and some or all of the submitted Talks could be merged into a single, stronger Talk. In this case, they will reject one and accept the other, suggesting such a merger. If this is not acceptable, please contact the Talks Chair to discuss the situation.
- The jury is unclear what they would learn by attending the Talk. Vague Talks are a waste of attendee time. Your submission should provide insight on the Talk content.
Experience Presentation
If you are proposing an installation and also want to give a presentation about your it, look for this checkbox on the submission form: If your installation is accepted, are you willing to also give a presentation about your installation?
If your work is accepted for Emerging Technologies, the Studio, or Immersive Pavilion, your Experience Presentation will be evaluated based on the following criteria:
- Benefits to attendees
- How will presenting your technology, process, or project directly benefit attendees? What will they be learning and how can they apply what they learn to their own work?
- Level of technical innovation
- Does the project represent a new step forward in graphics and interactive technology?
If your work is not accepted, your Experience Presentation also will not be accepted. Giving an Experience Presentation is NOT a requirement for acceptance of your installation.
Education Talks
Accepted Education Talks can include case studies of exemplary interdisciplinary courses and projects, curricula, tools, and new courses at the leading edge of graphics and interactive techniques content. Most educators attending the conference focus on undergraduate education, but there also is interest in informal education, K-12 education, and graduate education. Education Talks are often scheduled as part of the Educator’s Forum at the conference. Submitters are asked to select education as a category when submitting to help identify works appropriate for this focus area.
Non-Disclosure Agreements
SIGGRAPH reviewers cannot sign non-disclosure agreements for submissions. For information on patents and confidentiality, see the Submissions FAQ.
