Parallel Programming

The course starts with a motivation for parallel programming and a classification of parallel architectures. It focuses first on parallelization for distributed memory architectures with MPI. It introduces the major concepts, e.g., point to point communication, collective operations, communicators, virutal topologies, non-blocking communication, single-sided communication and parallel IO. In addition, it covers the overall parallelization approach based on four phases, i.e., decomposition, assignment, orchestration and mapping. The next section presents dependence analysis as the major theoretical basis for parallelization. It introduces program transformations and discusses their profitability and safety based on data dependence analysis. The second major programming interface in the course is OpenMP for shared memory systems. This section covers most of the language concepts as well as proposed extensions. In the last part, the lecture presents novel programming interfaces, such as PGAS languages, threading building blocks, CUDA, OpenCL, and OpenACC.

At the end of the module students are able to create parallel programs in MPI and OpenMP. They understand the performance aspects of differenct parallelization strategies and can evaluate those parallelization strategies in the context of applications. They are able to apply data dependence analysis and program transformations. They can analyze and tune the performance of parallel applications.

IN2076 Advanced Computer Architecture

The different parallel programming models and parallelization techniques are introduced in the lecture. Voluntary short student presentations demonstrate the techniques in application areas. Within a central exercise session, assignments are presented and discussed. The students solve the assignments and submit the solutions which are checked for correctness. In the assigments the students apply the learned concepts to larger example programs.

Slides

- MPI: A Message-Passing Interface Standard (Language Standard)
- OpenMP: Open Application Program Interface (Language Standard)
- R. Allen, K. Kennedy: Optimizing Compilers for Modern Architectures, Morgan Kaufmann
- David E. Culler et.al.: Parallel Computer Architecture: A Hardware / Software Approach, Morgan Kaufmann

The exam takes the form of a 90 minutes written test. Questions allow to asses acquaintance with the concepts of parallel programming models, languages, and tools. Code snippets of sequential and parallel programs are given. Students apply their knowledge on dependence analysis and code transformations to these codes. Based on code snippets the students apply the learned parallel models to demonstrate their ability to evaluate different parallelization strategies, to parallelize code, and tune applications.