diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..99d8f1f --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 Andrej Karpathy + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/M1_README.md b/M1_README.md new file mode 100644 index 0000000..8cf7a61 --- /dev/null +++ b/M1_README.md @@ -0,0 +1,175 @@ +# ECE408 Final Project - GPT-2 + +## Milestone 1 + +## Table of Contents + +- [Introduction](#introduction) +- [Milestone Requirements](#milestone-requirements) +- [Implementation and Testing](#implementation-and-testing) +- [Code Submission and Grading](#code-submission-and-grading) +- [Profiling-and-Performance-Analysis](#profiling-and-performance-analysis) +- [Deliverables](#deliverables) + +## Introduction + +This milestone focuses on implementing a basic version of the GPT-2 model's forward pass kernels on the GPU using CUDA. You will focus on writing a simple functionally correct GPU kernel for each step of the forward pass without worrying about optimizing your code for performance. You may also try to get comfortable with using NVIDIA's profiling tools to identify performance bottlenecks and areas for optimization, though this will not be a requirement for this milestone. + +To understand the structure of the forward pass (as well as to understand the variable names), **read the `gpt2.cuh` file.** It has a lot of information that helps with demos as well. + +## Milestone Requirements + +For this milestone, you will need to write a basic parallel implementation for all forward pass kernels in the `kernels` folder. Before completing these kernels, you should make sure that you have a good conceptual understanding of the GPT-2 transformer architecture. For example, you should understand what each kernel does and the role that each plays in the forward pass of the GPT-2 model. We may ask you conceptual questions related to the GPT-2 architecture as part of this milestone's demo. (More information on the demo is in [README.md](README.md)). + +**Note:** Your implementation needs to include the host code that launches your kernels, but you do NOT need to allocate/copy memory for the inputs/outputs. This means that you may assume memory management and copying is done properly by the provided code outside of the "`xxxx_forward()`" functions, and all you need to write in them is code that launches your kernel(s). You **should not** make changes to the function signatures of the provided "`xxxx_forward()`" **host functions**, but you may modify the **kernel function signatures** (or add additional kernels that your host functions calls) as needed. + +**Important:** + - We expect you to make consistent and frequent pushes to your repository. This will help us understand and track your progress. + - You should also ensure that your code follows good coding conventions. For example, any unclear or unusual code/tricks should be explained with comments, and variable names should generally be descriptive or clear from context. Your grade will depend on your thorough understanding of the code and code clarity. + + +## Implementation and Testing + +### Testing Individual Kernels + +For this milestone, we provide a few different test scripts for you to verify the correctness of each of your forward pass kernels, as well as your overall inference output correctness. + + +#### output_verification_rand +The file `output_verification_rand.cu` contains some skeleton test code that allows you to quickly test the basic functionality of each kernel you wrote by feeding in randomly generated inputs and comparing the outputs of your kernels to the provided CPU kernels. We encourage you to modify this for your own debugging purposes. + +In this script, the tests for each kernel are run independently, which means the test result of each kernel only depends on the correctness of that kernel, and errors in one kernel does not affect the correctness of the other kernels. + +**NOTE:** This file is provided to quickly get you started on debugging your individual kernel implementations. You need to make some small changes to the script before you start (marked with "TODO: "). We recommend using a lenient error_threshold when you first start testing, since this test script only compares outputs from the provided reference **CPU** code to that of the **GPU** code. (Why one shouldn't naively set the error threshold to some super small value or 0 to ensure "correctness" is left to the reader as an exercise). You should **NOT** try to use this test script to guarantee the *full correctness* of your kernels. + +To use this test script, run the command + + make output_verification_rand + +then you can run the test script by uncommenting the line `srun ./output_verification_rand` in the `verify.slurm` file and then running. + + sbatch verify.slurm + +The outputs of the testing script will be saved in the `verify_output.out` file. + +To clean, run + + make clean + +This will remove all the files generated during the compilation and execution process. + +#### test_gpt2_kernels + +The script `test_gpt2_kernels.cu` tests the functionality of your kernels by simulating a forward pass and comparing the outputs of your kernels at each step to the outputs of our gold forward pass kernel implementations (which runs on the GPU). + +To use this test script, run the command + + make test_gpt2_kernels + +then you can run the test script by uncommenting the line `srun ./test_gpt2_kernels` in the `verify.slurm` file and then running. + + sbatch verify.slurm + +The outputs of the testing script will be saved in the `verify_output.out` file. + +### Overall Inference Output Testing + +After you are (somewhat) confident with all of your kernel implementations, you can run the provided full inference test script to verify the correctness of your GPT-2 forward pass implementation. + +Since the full forward pass requires all of the implemented kernels to be correct, you should only run this test after you have verified the correctness of all of your kernels. + +To build your implemented kernels and verify the output of the full forward pass of the GPT-2 model, run the command + + make test_gpt2 + +Then, make sure the line `srun ./test_gpt2` is uncommented in the `verify.slurm` file and run + + sbatch verify.slurm + +The outputs of the testing script will be saved in the `verify_output.out` file. + +### Talk to the GPT2 You Wrote! + +Once you have implemented the forward pass correctly, you will be able to talk* to the GPT-2 model you wrote! You can input text into the model and have it complete the text for you. To do this, run the command + + make next_token_generation + +Then, go into `generate_tokens.slurm` and edit the text you want the model to complete. After that, run + + sbatch generate_tokens.slurm + +Once the job has completed, you can then find the model outputs in `generate_tokens_output.out`! + +*\*talk: voice recognition not included, and no fancy chatbot capabilities, but by inputting text and having the model complete it, you can still technically talk to the GPT-2 model you wrote!* + +*Also note that the tokenizer is not fully implemented. Prompts with simpler words/vocabulary should work fine, but more uncommon words/slangs or complex symbols in the input may result in non-ideal output.* + +## Code Submission and Grading + +To submit your code, push to the main branch of your group's repository. +Only code in the `kernels` folder will be graded. + +The grade breakdown for the coding portion of this milestone is as follows: + +| Kernel | Weight | +|-------------------------------|------- | +| Attention (including Softmax) | 3% | +| Encoder | 1% | +| LayerNorm | 2% | +| MatMul | 2% | +| Residual | 1% | +| GELU | 1% | + +*Total: 10% of project grade* + +## Profiling and Performance Analysis + +You may use NVIDIA's profiling tools to identify performance bottlenecks and areas for optimization. Although profiling is **not required** in this milestone, it is highly recommended to start profiling your baseline implementation once you're done implementing the kernels. You will be asked to profile your baseline implementation in Milestone 2. + +Given that you have implemented a MVP (Minimum Viable Product) of the GPT2 forward pass, you should be able to profile the performance of your implementation and identify bottlenecks and potential areas for optimization. + +**Before you do any profiling, make sure your implementation is functionally correct. Also make sure you do not have any memory errors by running *compute-sanitizer*.** +This tool is already installed and available in the delta environment. To use it, simply add `compute-sanitizer` after `srun`. For more information, see the [NVIDIA compute-sanitizer documentation](https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html). + +For this course, we're using Delta's A40x4 nodes, where each node has four A40 GPUs. Since each job submitted to Delta usually uses only one GPU, a single node can handle up to four jobs simultaneously. However, when profiling your code, it's important to avoid external interference. **Therefore, you need to ensure that you're adding the `perf,nvperf` constraints in your slurm file to ensure exclusive access to the node.** The trade-off is longer wait times if the cluster is busy. It's recommended to remove `perf,nvperf` when testing the correctness of your code, and include it only during profiling. + +The top of your slurm script should look like this when profiling: + +```bash +#!/bin/bash +#SBATCH --job-name="verify" +... +#SBATCH --constraint="projects,perf,nvperf" +... +``` + + +***System level profiling using Nsight-Systems*** + +`nsys` (Nsight Systems) profiles the execution at the system/application level. You can generate a profile using the `nsys` command provided in the `verify.slurm` file. + +You can also download the Nsight Systems application, where you can visualize the timeline of your application's execution. Download the `.nsys-rep` file generated from delta onto your local system and open it in the Nsight Systems app to visualize the timeline. + +You can find more information about nsys in the [Nsight Systems Documentation](https://docs.nvidia.com/nsight-systems/UserGuide/#cli-profiling). + +***Kernel level profiling using Nsight-Compute*** + +Please not that Nsight-Systems does not give you detailed kernel level performance metrics. For that, we will need to use `ncu` (Nsight-Compute). + +You will need to install NVIDIA NSight Compute on your own machine. It can be downloaded from NVIDIA's [website](https://developer.nvidia.com/nsight-compute) as a standalone application. + +After installing NSight Compute, you can profile your code by running the command provided in the slurm file. + +You can open the generated `.ncu-rep` file in the Nsight Compute application. Nsight Compute provides detailed information about the performance of each kernel, including compute latency/throughput metrics, memory workload analysis, and more. + +## Deliverables + + +| Step | Deliverables | +| ---- | ------------------------------------------ | +| 1 | Implemented forward pass kernels in the `kernels` folder | +| 2 | Knowledge of the GPT-2 architecture | +| 3 | M1 Demo Meeting | + + +We are excited to hear from you soon! diff --git a/M2_README.md b/M2_README.md new file mode 100644 index 0000000..2483731 --- /dev/null +++ b/M2_README.md @@ -0,0 +1,261 @@ +# ECE408 Final Project - GPT-2 + +## Milestone 2 + +## Table of Contents + +- [Introduction](#introduction) +- [Milestone Requirements](#milestone-requirements) +- [Implementation and Testing](#implementation-and-testing) +- [Performance Analysis/Profiling](#performance-analysisprofiling) +- [Profiling Report and Optimization Proposal](#profiling-report-and-optimization-proposal) +- [Submission and Grading](#submission-and-grading) +- [Deliverables](#deliverables) +- [Additional Details/Hints for Optimizations](#additional-detailshints-for-optimizations) + +## Introduction + +This milestone focuses on profiling and optimizing some of the GPU kernels from Milestone 1, and using your profiling results to guide you to propose some of your own optimizations for Milestone 3. + +## Milestone Requirements + +For this milestone, you will need to: +- Complete the below list of required optimizations. +- Profile your M1 baseline kernel implementations as well as all M2 required kernel optimizations. +- Present your profiling findings in the form of a report. +- Analyze your results to motivate additional optimizations outside of the list currently supported for the next Milestone, and propose them in your report. + +### Milestone 2 Required Optimizations: +| Optimization | Description | +| ------------ | ----------- | +| req_0 | Joint Shared Memory and Register Tiling for Matrix Multiplication | +| req_1 | Tensor Core for Matrix Multiplication | +| req_2 | Utilize cuBLAS | +| req_3 | Reduction | + +**Note:** You may **NOT** use cuBLAS in req_0, req_1, or req_3. + +## Implementation and Testing + +Refer to M1_README.md for instructions on how to use the provided testing scripts; you may use the same ones from Milestone 1. Note that for `req_1` and `req_2` you will do some of your calculations using lower precision, so your errors may be higher than in Milestone 1. + +## Performance Analysis/Profiling + +In this milestone, You are **required** to use NVIDIA's profiling tools to identify performance bottlenecks and areas for optimization. + +Given that you have implemented a MVP (Minimum Viable Product) of the GPT-2 forward pass, and several additional optimizations, you should be able to profile the performance of your implementation and identify bottlenecks in your baseline implementation and draw comparisons with your implemented optimizations. + +**Before you do any profiling, make sure your implementation is functionally correct. Also make sure you do not have any memory errors by running *compute-sanitizer*.** +This tool is already installed and available in the delta environment. To use it, simply add `compute-sanitizer` after `srun`. For more information, see the [NVIDIA compute-sanitizer documentation](https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html). + +For this course, we're using Delta's A40x4 nodes, where each node has four A40 GPUs. Since each job submitted to Delta usually uses only one GPU, a single node can handle up to four jobs simultaneously. However, when profiling your code, it's important to avoid external interference. **Therefore, you need to ensure that you're adding the `perf,nvperf` constraints in your slurm file to ensure exclusive access to the node.** The trade-off is longer wait times if the cluster is busy. It's recommended to remove `perf,nvperf` when testing the correctness of your code, and include it only during profiling. + +The top of your slurm script should look like this when profiling: + +```bash +#!/bin/bash +#SBATCH --job-name="verify" +... +#SBATCH --constraint="projects,perf,nvperf" +... +``` + +***System level profiling using Nsight-Systems*** + +`nsys` (Nsight Systems) profiles the execution at the system/application level. You can generate a profile using the `nsys` command provided in the slurm file. + +After running the command, you should see something that looks like the following (but not identical) in the generated `..._nsys_prof.out` file: + +```bash +...... + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ------------ ------------- -------- ----------- ----------- --------------------- + 99.9 351,122,724,860 3,519 99,779,120.4 100,089,303.0 2,855 100,130,281 5,413,528.2 poll + 0.1 283,382,530 925 306,359.5 14,207.0 1,051 20,208,549 1,050,067.9 ioctl + ...... + 0.0 1,913 1 1,913.0 1,913.0 1,913 1,913 0.0 bind + +[5/8] Executing 'cudaapisum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ------------ ----------- -------- ----------- ------------ ---------------------- + ...... + +[6/8] Executing 'gpukernsum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) GridXYZ BlockXYZ Name + -------- --------------- --------- ------------ ------------ ---------- ---------- ----------- --------------- -------------- ---------------------------------------------------------------------------------------- + ...... + +[7/8] Executing 'gpumemtimesum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ------------- ------------- ----------- ----------- ------------ ------------------ + ...... + +[8/8] Executing 'gpumemsizesum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- --------- ----------- ------------------ + ...... + +``` +The CUDA API Statistics section shows the CUDA API calls that are executed. The CUDA Kernel Statistics lists all the kernels that were executed during the profiling session. There are also more details on the CUDA memory operations (CudaMemcpy) listed. There are columns corresponding to percentage of time consumed, total time, number of calls, and average/min/max time of those calls. + +In the Nsight Systems application, you can visualize the timeline of your application's execution. Download the `.nsys-rep` file generated onto your local system and open it in the Nsight Systems app to visualize the timeline. + +You can find more information about nsys in the [Nsight Systems Documentation](https://docs.nvidia.com/nsight-systems/UserGuide/#cli-profiling). + +***Kernel level profiling using Nsight-Compute*** + +Nsight-Systems does not give you detailed kernel level performance metrics. For that, we will need to use `ncu` (Nsight-Compute). + +You will need to install NVIDIA Nsight Compute on your own machine. It can be downloaded from NVIDIA's [website](https://developer.nvidia.com/nsight-compute) as a standalone application. + +After installing Nsight Compute, you can profile your code by running the command provided in the slurm file. + +You can open the generated `.ncu-rep` file in the Nsight Compute application. Nsight Compute provides detailed information about the performance of each kernel, including compute latency/throughput metrics, memory workload analysis, and more. + +**Note**: Although Nsight Compute provides some useful analysis information based on the profiling data and may make some recommendations for optimization, it is important for you to fully understand exactly what causes the performance inefficiencies and why the suggestions Nsight Compute gave you are significant (*or why not*). + +## Profiling Report and Optimization Proposal + +After performing both system-level profiling using Nsight-Systems and kernel-level profiling using Nsight-Compute, your team will need to submit a report detailing your profiling results. +There is no strict visual structure requirement for this report, but your report should touch on at least the following topics: + +- **Analysis of Baseline Implementation** + - What are the most time-consuming kernels in your baseline implementation? + - How's your baseline kernels' memory access pattern? + - Some relevant metrics can be helpful. + +- **Performance Improvements from Required Optimizations** + - What's the performance difference between baseline and each optimization you implemented in this Milestone? + - Analyze why each optimization was effective (or why it wasn't) + - Detailed metrics can be used to support your analysis. + +- **Pretty Pictures and Data** + - We'd really appreciate some nice charts or graphs of performance metrics! (annotate them and use them to supplement your analysis instead of a raw screenshot would be even better) + - Comparison tables for things like execution time can be helpful too. + +- **Interesting Findings** + - Anything you find interesting and/or surprising from the profiling results. + +When writing this report, you should make full use of the detailed metrics provided by Nsight-Systems and Nsight-Compute. + +You should be detailed and in-depth about your findings and comparisons from profiling, **do NOT** just copy some metrics from Nsight, explain what those metrics are, and call it a day! + +This report does not have a length requirement. A long report packed with different metrics and surface-level analysis is **not** what we're looking for (*that's the job of the Nsight tools*). Your job is to interpret the data and provide **concise yet insightful conclusions**. This means we prefer quality over quantity, and some good insights on only a couple kernels is better than a bunch of surface-level analysis/boring metrics on every kernel. + +### Optimization Proposal +The logical next step after profiling your code thinking critically and coming up with you own optimizations. Following your profiling sections in your report, you will need to write up an optimization proposal section detailing how you plan on further optimizing the existing GPU forward pass kernels. + +Note that Milestone 3 will comprise of a list of required optimizations, as well as a set of additional optimizations that you will select a subset of to implement for credit, based on what interests you. The tentative list of all Milestone 3 requirements is shown below. + +Your proposal should contain around 2-4 additional optimizations outside of the list for Milestone 3 that you wish to add to the additional optimizations section. You may come up with your own ideas, analyze your profiling results, extend/expand an existing optimization (in a meaningful/nontrivial way), apply concepts from lecture, and/or search online for optimization ideas. Here are some ideas/options (*that aren't necessarily the easiest/best*): Multi-GPU Support, NVIDIA CUTLASS, a new algorithm to optimize a specific kernel, etc. + +Note that these optimizations may **not** already be one of the M3 optimizations that you can implement in the next milestone, but if your optimization is proposed clearly, we will add to the following list and assign it a point value. The M3 optimizations are listed below for your reference. + +You should provide a detailed description of the proposed optimizations, how you plan to implement them, and why you think they will be beneficial. Grading of this proposal will be based on effort, detail, and feasibility of the proposed optimizations. + +If your proposal is approved, it will be added to the pool of eligible Additional Optimizations. While you are highly encouraged to implement your own proposed ideas in Milestone 3, you will have the flexibility to pivot and choose from the standard Additional Optimizations list below if needed. + +#### Tentative Milestone 3 Required Optimizations (NOT for this milestone, cannot be proposed): + +| Optimization | Description | +| ------------ | ----------- | +| req_4 | Flash Attention | +| req_5 | Local/Windowed Attention | +| req_6 | KV-Caching | + +#### Tentative Milestone 3 Additional Optimizations (NOT for this milestone, cannot be proposed): + +| Optimization | Description | +| ------------ | ----------- | +| op_7 | Configuration Sweep/Optimization | +| op_8 | Constant Memory | +| op_9 | `__restrict__` | +| op_10 | Split-K | +| op_11 | Shared Memory Padding | +| op_12 | Data Swizzling | +| op_13 | Host Allocation Data Alignment | + +## Submission and Grading + +### Non-Code Submission +To submit your profiling report and optimization proposal, combine them as **1 PDF file** named `[team_name]_M2_Profiling_Report.pdf`, where `[team_name]` is your team's name and **submit to the Gradescope assignment**. Only one member from each team must submit, and ensure that the name of your team is somewhere on the first page. There is no strict format or page requirement for them as it will be manually graded, but make sure the file is typed (not hand-written) and correctly named. + +### Code Submission + +Your code should be submitted in the following manner: + ``` + sp26_ece408_[team_name] + ├── kernels + │ ├── attention.cuh + │ ├── encoder.cuh + │ ├── gelu.cuh + │ ├── matmul.cuh + │ └── ... [all unmodified kernels from M1] + ├── kernels_req_0 + │ ├── matmul.cuh [with the req_0 optimization, register tiling] + │ └── ... [any other files to replace in kernels/ for req_0] + ├── kernels_req_1 + │ ├── matmul.cuh [with the req_1 optimization, tensor core] + │ └── ... [any other files to replace in kernels/ for req_1] + ├── kernels_req_2 + │ ├── ... [any files to replace in kernels/ for req_2] + │ └── ... [any files to replace in kernels/ for req_2] + ├── kernels_req_3 + │ ├── ... [any files to replace in kernels/ for req_3] + │ └── ... [any files to replace in kernels/ for req_3] + ├── gpt2.cuh + ├── Makefile + ├── README.md + └── ... + ``` +You will submit each of your optimized kernels in separate folders named `kernels_req_x` where `x` is `0`, `1`, `2`, or `3`. During grading, when grading optimization `req_x`, we will copy any files from the `kernels_req_x` folder into the `kernels` folder, replacing any files with the same name. E.g., for `req_0`, `kernels_req_0/matmul.cuh` with the register tiling optimization will replace `kernels/matmul.cuh`, and those kernels will be run. + +For each optimization, you are expected to apply it everywhere it is applicable, within reason. If you are missing an expected file for an optimization, or it is not implemented as expected (manually checked), you will lose points on that optimization. It is your job as a team to figure out which kernels each optimization applies to, and include those files in the corresponding folder. You may however apply optimizations to additional kernels we did not expect; in fact we encourage you to do so. + +The grade breakdown for the coding portion of this milestone is as follows: + +| Optimization | Weight | +|------------------------------- |------- | +| req_0: Register Tiling | 6% | +| req_1: Tensor Core | 5% | +| req_2: Utilize cuBLAS | 4% | +| req_3: Reduction | 3% | + +*Total: 18% of project grade* + +Please note that you will only receive full credit for each optimization if it is implemented correctly. Just like the lab assignments, there will **not** be partial credit for incorrectly implemented/non-functioning code. + + +## Deliverables + +| Step | Deliverables | +| ---- | ------------------------------------------ | +| 1 | Implemented M2 Required optimizations | +| 2 | Profiling Results and Mini Report | +| 3 | Optimization Proposal | +| 4 | M2 Demo Meeting | + +## Additional Details/Hints for Optimizations + +### req_0: Joint Register and Shared Memory Tiling for Matrix Multiplication + +There will be a lecture on advanced optimizations (Lecture 16), which will cover an overview of joint register and shared memory register tiling. Additionally, you can refer to [this](https://lumetta.web.engr.illinois.edu/508/slides/lecture4.pdf) ECE 508 lecture slide on the concept. **You are required to implement this optimization in all kernels where its use is reasonable.** As a hint, there is more than 1 kernel where register tiling can be reasonably used. + + +### req_1: Tensor Core for Matrix Multiplication +There will be a lecture on advanced optimizations (Lecture 16), which will cover an overview of using Tensor Cores for matrix multiplication. Note that you are **required** to implement tensor core with *TF32* precision. You may refer to the advanced optimizations lecture, as well as online NVIDIA documentation/blog posts for more information. **You are required to implement tensor core in all kernels where its use is reasonable.** As a hint, there is more than 1 kernel where tensor core can be reasonably used. + + +### req_2: Utilize cuBLAS + +Make sure you understand how to properly setup and use cuBLAS. Throwing in a single function call to cuBLAS without understanding how it works may not earn you full points (and probably won't work). The [NVIDIA cuBLAS documentation](https://docs.nvidia.com/cuda/cublas/#using-the-cublas-api) may be helpful. **You are required to implement cuBLAS in all kernels where its use is reasonable.** As a hint, there is more than 1 kernel where cuBLAS can be reasonably used. + + +### req_3: Reduction + +Here reduction refers to the algorithm covered in lecture 15 (and lab 6). **In this project, you are required to apply the reduction algorithm you learned in lab to all kernels where its use is reasonable.** Think about in which kernel you run a reduction-type calculation, combining several values into a single value where the operation is associative and commutative. As a hint, there is more than 1 kernel where reduction can be reasonably used. diff --git a/M3_README.md b/M3_README.md new file mode 100644 index 0000000..560f617 --- /dev/null +++ b/M3_README.md @@ -0,0 +1,321 @@ +# ECE408 Final Project - GPT-2 + +## Milestone 3 + +## Table of Contents + +* [Introduction](#introduction) +* [Milestone Requirements](#milestone-requirements) +* [Implementation and Testing](#implementation-and-testing) +* [Performance Analysis and Profiling ](#performance-analysis-and-profiling) +* [Final Report](#final-report) +* [Deliverables](#deliverables) +* [Submission and Grading](#submission-and-grading) +* [Additional Details/Hints for Optimizations](#additional-detailshints-for-optimizations) + + [Required Optimizations Details](#required-optimizations-details) + + [Additional Optimizations Details](#additional-optimizations-details) + +## Introduction + +This milestone focuses on further optimizing the GPU kernels from Milestone 2. Your aim is to create the world's best possible implementation of the GPT-2 model's forward pass kernels on the GPU. + +This is where you let your imaginations fly and CUDA/ML knowledge shine! + +You will also profile your optimizations and present your findings in a final report, one detailed enough for you to remember your work forever! + +## Milestone Requirements + +This milestone is worth a total of 50% of your project grade. The coding portion is split into two mandatory categories: **Required Optimizations (20%)** and **Additional Optimizations (10%)**. + +For full credit on this milestone, you will need to: +- Complete the below list of required optimizations +- Select and implement a subset of the **Additional Optimizations** such that their combined point values equal at least 10%. +- Profiling: + - Find each individual required and additional optimization's effectiveness through various profiling metrics + - Determine **specific reasons** for speedups/slowdowns/no effect + - Describe how different optimizations work together (or against each other) + - Compare the final optimized implementation with baseline kernels from Milestone 1 +- Present findings: + - Final Report + +### Required optimizations (20% Total): + +You **must** implement all of the following optimizations: + +| Optimization | Description | Weight | +| ------------ | ----------- | ------ | +| req_4 | Flash Attention | 6% | +| req_5 | Local/Windowed Attention | 2% | +| req_6 | KV-Caching | 12% | + +### Additional optimizations (10%+ Total): + +To earn the remaining 10% of your coding grade, you must mix and match from the pool of additional optimizations below. This list includes standard advanced optimizations as well as excellent ideas proposed by teams during Milestone 2. + +You may pick any combination of the following, so long as the weights add up to at least 10%. *(Note: If your selected optimizations exceed 10%, the extra points may be considered for extra credit, up to 10% max when combined with leaderboard extra credit).* + +| Optimization | Description | Weight | +| ------------ | ----------- | ------ | +| op_7 | Configuration Sweep/Optimization | 3% | +| op_8 | Constant Memory | 2% | +| op_9 | `__restrict__` | 1% | +| op_10 | Split-K | 3% | +| op_11 | Data Swizzling | 3% | +| op_12 | Shared Memory Padding | 3% | +| op_13 | Host Allocation Data Alignment | 2% | +| op_14 | Basic CUTLASS Device GEMM | 3% | +| op_15 | Advanced Tensor Core Math (CuTe or Raw PTX) | 5% | +| op_16 | Advanced CUTLASS with Fused Epilogues | 4% | +| op_17 | Asynchronous Memory Pipelines (`cp.async`) | 4% | +| op_18 | Thread Block Rasterization / Scheduling | 2% | + +**CUTLASS Mutually Exclusive Rule**: You may only choose **ONE** of the CUTLASS/Tensor Core optimizations (`op_14`, `op_15`, or `op_16`) to implement for credit. You cannot stack these for combined points. Also, you may not use CUTLASS or any other NVIDIA libraries to implement the other optimizations for you. + +**Important**: Please see the additional details and hints provided at the end of this document before beginning your implementations. + +## Implementation and Testing + +For this milestone, most optimizations will be graded on accuracy/correctness using the existing verification scripts (`test_gpt2_kernels.cu` or `test_gpt2.cu`). + +However, because you are implementing structural architectural changes (like `req_5: Local/Windowed Attention`) and selecting your own mix of additional optimizations, your group is ultimately responsible for demonstrating the correctness of any optimizations that intentionally cause deviations from our provided op-level and end-to-end solution values. + +In other words, if a chosen optimization changes the math such that it fails tests in either `test_gpt2_kernels.cu` or `test_gpt2.cu`, you will need to develop a sufficient verification method (for example, using the *Perplexity* scoring system to evaluate the model's language modeling accuracy) and justify in your final report why the optimization is implemented correctly. Without this justification, you **may NOT** receive full credit. For optimizations that do not cause deviations, the existing verification scripts are sufficient. + +Since `req_5` (Local/Windowed Attention) changes the attention mechanism from full attention to local/windowed attention, it is expected that this optimization will cause deviations from our provided solution values. See the additional details section for more information on how to verify this specific optimization. + +## Performance Analysis and Profiling + +Refer to [M2_README.md](M2_README.md) for profiling information. + +## Final Report + +After performing both system-level profiling and kernel-level profiling, your team will need to submit a final report detailing the following: +- **Implementation Details** + - For each optimization (both required and your chosen additional ones), describe how you implemented it. + - If your chosen additional optimizations did not yield performance improvements, explain why you think that is the case. + - Explain how you verified each implementation's correctness (especially if the final model outputs deviated from the baseline). + - Discuss any challenges you faced during implementation and how you overcame them. + + +- **Performance Analysis of Milestone 3 Optimizations** + - Compare the performance of your final optimized kernels with the baseline implementations from Milestone 1. + - Be sure to cover **all** new Milestone 3 optimizations (required and additional). + - Analyze why each optimization was effective (or why it wasn't). + - What bottlenecks are addressed by your final version of each kernel? + - Logical conclusions should be drawn in addition to just reporting metrics. + - Detailed metrics and graphs can be used to support your analysis. + - End-to-end performance comparisons should also be included here. + +- **Pretty Pictures and Data** + - We'd really appreciate some nice charts or graphs of performance metrics! (annotate them and use them to supplement your analysis instead of a raw screenshot would be even better) + - Comparison tables for things like execution time can be helpful too. + +- **Interesting Findings** + - Anything you find interesting and/or surprising from the profiling results. + +When writing this report, you should make full use of the detailed metrics provided by Nsight-Systems and Nsight-Compute. + +You should be detailed and in-depth about your findings and comparisons from profiling, **do NOT** just copy some metrics from Nsight, explain what those metrics are, and call it a day! Make sure to explain and provide insights on **why** the metrics look the way they do, and what implementations decisions led to those metrics. + +This report does not have a length requirement. A long report packed with different metrics and surface-level analysis is **not** what we're looking for (*that's the job of the Nsight tools*). Your job is to interpret the data and provide **concise yet insightful conclusions**. This means we prefer quality over quantity, and some good insights on only a couple kernels is better than a bunch of surface-level analysis/boring metrics on every kernel. + +Your report should be a document that records all high level design decisions, optimizations, and performance analysis done throughout the entire GPT Final Project. The report should be written in a way that is accessible to a general audience, but also contains details that would allow you (or anyone else) to reproduce your results if you look back to this project 5 years later. + +## Deliverables + +| Step | Deliverables | +| ---- | ------------------------------------------ | +| 1 | Implemented M3 Required Optimizations | +| 2 | Implemented M3 Additional Optimizations | +| 3 | Profiling and Performance Analysis | +| 4 | Milestone 3 Demo | +| 5 | Final Report | + +## Submission and Grading + +To submit your final report, push it as a **PDF file** to the root directory of your GitHub main branch named `[team_name]_Final_Report.pdf`, where `[team_name]` is your team's name. There is no strict format requirement for them as it will be manually graded, but make sure the file is typed (not hand-written) and correctly named. + +### Code Submission + +Your code should be submitted in the main branch of your repository. To ensure the autograder can correctly interface with your mix-and-match selections, you must organize your code into specific folders. + +Required optimizations must be placed in folders named `kernels_req_x`. Your chosen additional optimizations must be placed in folders named `kernels_op_x`. + +For example, if your team chose to complete Configuration Sweeping (`op_7`), Constant Memory (`op_8`), Split-K (`op_10`), and Host Allocation Data Alignment (`op_13`) to reach your 10% requirement (with 1% extra credit), your directory structure should look exactly like this: +``` + sp26_ece408_[team_name] + ├── kernels + │ ├── attention.cuh + │ ├── encoder.cuh + │ ├── gelu.cuh + │ ├── matmul.cuh + │ └── ... [unmodified kernels from M1] + ├── kernels_req_4 + │ └── attention.cuh [with Flash Attention] + ├── kernels_req_5 + │ └── local_attention.cuh [with Local/Windowed Attention] + ├── kernels_req_6 + │ └── ... [with KV-Caching] + ├── kernels_op_7 + │ └── [configuration sweep script] + ├── kernels_op_8 + │ └── ... [with constant memory] + ├── kernels_op_10 + │ └── ... [with Split-K] + ├── kernels_op_13 + │ └── gpt2.cuh [modified to have aligned allocations] + ├── gpt2.cuh + ├── Makefile + ├── README.md + └── ... +``` + +**Important**: Any incorrect/non-functioning implementations will receive 0 correctness points. However, you may still earn points for your understanding and analysis of the optimization in your final report even if the implementation is incorrect/non-functioning. + + +## Additional Details/Hints for Optimizations + +### Required Optimizations Details + +**Note:** Lecture 23 goes through all of the required optimizations for this project in great detail. We highly recommend reviewing this lecture before starting your implementations! + +#### req_4: Flash Attention + +The FlashAttention series of papers proposed a monumental IO-aware attention optimization that is now widely used in the industry. The algorithm enables tiling of the attention matrix to address the memory bandwidth bottleneck in attention. You may choose to implement either the basic FlashAttention algorithm, outlined in the original [FlashAttention paper](https://arxiv.org/pdf/2205.14135), **or** the improved version, outlined in the [FlashAttention2 paper](https://tridao.me/publications/flash2/flash2.pdf). + +**Note:** Both FlashAttention and FlashAttention2 are long papers that may take a while to read fully, so we recommend focusing on the following sections: + - If you choose to implement the basic FlashAttention: Introduction, Section 2.2 (Standard Attention Implementation), Section 3.1 (**Skip** discussions of Recomputation) + - If you choose to implement FlashAttention2: Introduction, Section 2.2 (Standard Attention Implementation), Section 2.3.1 (Basic FlashAttention Algorithm), Section 3.1.1 (FlashAttention2 Algorithm). + + *If you are using the FlashAttention2 paper on arxiv.org, there may be a minor mistake in the pseudo code outlined in section 3.1.1. Please use the paper link provided above to ensure you have the correct version.* + +The papers have already provided pseudo code for the algorithms, your job is to translate them into efficient CUDA code. + +#### req_5: Local/Windowed Attention + +We have provided a reference CPU implementation in `cpu_kernels/local_attention.cuh` for this optimization. Make sure you understand what exactly local/windowed attention is based on the provided CPU code before starting this optimization. +Additionally, we have provided a basic skeleton test script (*which you need to modify yourself*) for this optimization (`local_attn_verify.cu` and `local_attn_verify.slurm`). You may refer to these files when implementing and verifying your local/windowed attention optimization. +**Important:** For grading and verification purposes, you need to submit using a **window size of 128** (as shown in `local_attention.cuh`). However, once you have verified your implementation with a window size of 128 using our provided test data, feel free to experiment with different window sizes and include your findings in the final report. +**Note:** Make sure to use `make local_attn_verify` to compile your code base instead of simply using `make` to ensure your modifications to the local attention verification script is compiled correctly. + +#### req_6: KV-Caching + +To understand KV-Caching, you first need to understand the bottleneck of autoregressive text generation. When a Large Language Model generates text, it does so one token at a time. To generate the 100th token, the model must compute the Attention scores for the new token against the previous 99 tokens. In a naive implementation, the model recalculates the Key (K) and Value (V) tensors for all 99 previous tokens from scratch every single time it generates a new word. + +KV-Caching eliminates this massive redundant computation. Instead of throwing away the Key and Value matrices after a token is generated, you will implement a system to persistently store ("cache") them in GPU memory. + +During the generation phase, your code will only compute the K and V for the single newest token, append it to the cache, and perform the attention math using the newly updated history. You will need to carefully manage memory allocations to hold this growing cache across multiple iterations of the forward pass without crashing or overwriting data. + +### Additional Optimizations Details + +#### op_7: Configuration Sweep/Optimization + +For this optimization, create a script (python, bash, or any language of your choice) to systematically sweep through different configurations like block sizes, thread counts, loop unrolling, etc. +*Note for loop unrolling: Loop unrolling can technically be applied to any loop in your code, but you should understand where you should focus on and why. Make sure to perform thorough profiling after implementing this optimization and be able to directly point out the effects of loop unrolling.* + +Make sure you at least provide why and how you are sweeping each configuration, what you expected before the sweep, and what you found after the sweep. You should try a sufficient amount of configurations to ensure you are getting the best performance from this optimization. + +In short, you should be able to answer the following questions: +- How does your script sweep through different configurations? +- What configurations did you try? +- Why did you choose to focus on these specific parameters? +- What are your findings? Include expectations before and analysis/conclusion after. + +**Make sure to submit your sweeping script by pushing it to the main branch of your group's repository.** Note that we will not grade this script. However, we will manually check that you have submitted a working sweeping script in the `kernels_op_7` folder. You should also refer this script in your final report's when discussion this optimization. + +Note that there is not a minimum number of configurations you need to try, but make sure you try enough configurations to ensure you are getting the best performance from your kernels, and sufficient justifications of your choices and analysis in your final report will earn you full points for this optimization. + +#### op_8: Constant Memory + +Think about what data you can/should store in constant memory. Review the lecture covering constant memory if needed. Make sure to perform thorough profiling after implementing this optimization to see how it is actually improving performance (*or explain why it's not*). + +You may choose which kernel(s) to apply this optimization to, and providing sufficient justifications of your choices and analysis in your final report will earn you full points for this optimization. + +#### op_9: `__restrict__` + +Make sure you understand what `__restrict__` does and how it can help improve performance before starting. You should also be able to explain why (and where) you are using `__restrict__` in your code. + +You may choose which kernel(s) to apply this optimization to, and providing sufficient justifications of your choices and analysis in your final report will earn you full points for this optimization. + +#### op_10: Split-K + +Split-K is a technique that splits the K dimension of a GEMM operation into smaller chunks, allowing for better parallelism and memory access patterns. This can lead to improved performance, especially for certain matrix sizes. For more details on Split-K, you may refer to the advanced optimizations lecture (Lecture 16). + +You may choose which kernel(s) to apply this optimization to. Correct implementation of this optimization to **at least 1 kernel** will earn you full points for this optimization, but you **need** to perform thorough profiling and provide detailed analysis in your final report on how this optimization is affecting performance and justify the implementation choices you made. + +#### op_11: Data Swizzling (Shared Memory) + +NVIDIA GPUs divide shared memory into 32 distinct "banks." You can think of these banks like checkout lanes at a grocery store. If 32 threads in a warp try to access data from 32 different lanes, the transactions happen instantly in parallel. But if multiple threads try to read from the *same* lane (a "bank conflict"), the hardware forces them to wait in line, serializing the requests and tanking your memory bandwidth. This often happens when reading data column-by-column from a standard memory tile. + +"Swizzling" is a compute-based trick to fix this without changing the size of your array. By using a reordering operation, such as a bitwise XOR operation (`^`) between the row and column indices when calculating your shared memory addresses, you dynamically scramble (or "swizzle") the data layout. + +This mathematical permutation ensures that elements of the same column are no longer stacked directly on top of each other in the same memory bank. When threads go to read that column, they apply the exact same XOR math to find the data, pulling from 32 distinct banks in perfect parallel. + +#### op_12: Shared Memory Padding + +This optimization solves the exact same bank conflict problem described in Data Swizzling, but it fixes it using the memory footprint instead of mathematics. + +If the inner dimension of your 2D shared memory tile aligns perfectly with the number of hardware memory banks, any thread warp reading a column will trigger a massive bank conflict. Padding solves this by deliberately altering the memory stride. By adding "dummy" padding to the inner dimension of your shared memory allocation, you force consecutive elements of a column to shift into different memory banks. + +There is no single "correct" way to pad. Some implementations add a minimal amount of padding to slightly stagger the rows, while others significantly increase (or even double) the allocation footprint to drastically spread out the bank accesses. You are expected to explore different padding dimensions. + +Your goal is to use your profiling tools to find a strategy that successfully eliminates bank conflicts without wasting so much shared memory that you severely limit your Streaming Multiprocessor (SM) occupancy. + +#### op_13: Host Allocation Data Alignment + +When profiling your code in Nsight Compute (NCU), you might encounter a frustrating scenario: the tool reports "L2 sector global excess" (meaning your global memory reads are uncoalesced), even though you are absolutely certain your thread block indices and matrix dimensions are perfectly aligned. + +If your indexing math is perfect but the hardware still registers uncoalesced memory traffic, the problem often lies in *where the memory allocation actually starts*. The GPU fetches global memory in hardware-aligned chunks (typically 32-byte L2 cache lines). If the starting pointer of your data array lands in the middle of a cache line, a perfectly coalesced warp read will straddle two physical cache lines, forcing the GPU to fetch double the memory it actually needs. + +Take a close look at how the device pointers for the activation tensors are allocated in the host code. The codebase uses a single large memory allocation and mathematically partitions it into smaller chunks for the different tensors. Think about what happens to the starting memory address of tensor *N+1* if the total byte size of tensor *N* isn't perfectly aligned to a hardware-friendly boundary. + +To earn credit for this optimization, you must modify the host-side allocation logic to enforce strict memory alignment boundaries (hint: experiment with different alignment sizes) and prove the elimination of the NCU memory warnings in your final report. + +#### op_14: Basic CUTLASS Device GEMM + +*(Note: You may only select one CUTLASS optimization to implement for credit)*. + +CUTLASS is a collection of CUDA C++ template abstractions for implementing high-performance matrix multiplications. For this optimization, you will replace your standard GEMM or cuBLAS calls with CUTLASS's pre-defined device-level GEMM templates (e.g., `cutlass::gemm::device::Gemm`). + +To earn credit, you cannot simply copy-paste a default template. You must properly configure the `ThreadblockShape`, `WarpShape`, and `InstructionShape` parameters to specifically target the architecture of our A40 GPUs, and justify your parameter choices in your final report based on your profiling data. + +Check out the [CUTLASS Documentation Hub](https://docs.nvidia.com/cutlass/4.4.2/overview.html) and flip through the version of CUTLASS and different optimizations that you want to explore here! + +#### op_15: Advanced Tensor Core Math (CuTe or Raw PTX) + +*(Note: You may only select one CUTLASS/Tensor Core optimization to implement for credit. If you choose the raw PTX route, this rule still applies to prevent combining it with CUTLASS templates).* + +CUTLASS 3.x introduced **CuTe**, a powerful C++ template library for defining and manipulating hierarchical layouts of threads and data. While basic CUTLASS relies on pre-built `cutlass::gemm::device` templates, this optimization requires you to drop down a level of abstraction. + +For this optimization, you will bypass the standard `wmma` API to interact more directly with the hardware Tensor Cores. You may choose one of two paths: +1. **CUTLASS CuTe:** Utilize CuTe's `Tensor` and `Layout` objects to explicitly define how memory tiles map to threads and orchestrate the MMA atoms without writing assembly. + +2. **Raw PTX:** Use inline PTX instructions (specifically `ldmatrix` and `mma.sync`) to load data directly into the exact register layouts expected by the Tensor Cores, reducing Load/Store Unit (LSU) pressure and enabling shared memory swizzling. + +A good resource to begin learning CuTe may be the [CuTe Quick Start Guide](https://docs.nvidia.com/cutlass/latest/media/docs/cpp/cute/00_quickstart.html). + +To earn credit, your final report must include a detailed breakdown of your custom CuTe layouts OR your PTX instruction pipeline, explaining how your thread-to-data mapping works and why this approach is efficient for the A40 architecture. + +#### op_16: CUTLASS Fused Epilogue (Matmul + Bias + GeLU) + +*(Note: You may only select one CUTLASS optimization to implement for credit)*. + +Global memory bandwidth is a massive bottleneck in large language models. While basic kernels can often handle fusing a Bias addition to a Matmul, applying a non-linear activation function typically forces the GPU to write the intermediate results to global memory and launch a completely separate kernel just to apply the activation. + +For this optimization, you will configure a CUTLASS GEMM pipeline to include a **Fused Epilogue that incorporates the GeLU activation**. This allows the Streaming Multiprocessor to compute the Matmul, add the bias vector, and apply the complex GeLU mathematics directly in registers or shared memory *before* ever writing the final result back to global memory. + +You may either choose to use [GeLU Linear Combination](https://github.com/NVIDIA/cutlass/blob/main/include/cutlass/epilogue/thread/linear_combination_gelu.h) from Cutlass 2.x or [Epilogue Visitor Trees](https://developer.nvidia.com/blog/cutlass-3-x-orthogonal-reusable-and-composable-abstractions-for-gemm-kernel-design/) from Cutlass 3.x. + +To earn credit, your final report must include profiling data that explicitly demonstrates the elimination of the standalone GeLU kernel launch and the corresponding reduction in global memory traffic (DRAM read/writes) compared to your un-fused baseline implementation. + +#### op_17: Asynchronous Memory Pipelines (`cp.async`) + +In a standard kernel, loading from global to shared memory follows a strict path: Global -> Registers -> Shared. This consumes valuable register space and compute cycles. Ampere GPUs (like our A40s) introduced hardware support for asynchronous memory copies that bypass the register file entirely. + +For this optimization, utilize `cuda::pipeline` or raw `cp.async` PTX instructions to copy memory tiles directly from global DRAM into shared memory. By combining this with software pipelining, your Streaming Multiprocessors can perform math on one tile while asynchronously fetching the next tile in the background. Your report must demonstrate how this reduces register pressure and successfully overlaps memory fetches with computation. + +#### op_18: Thread Block Rasterization (L2 Cache) + +By default, the GPU schedules thread blocks linearly (row by row). If the matrix is large, by the time the GPU wraps around to start computing the next row of output tiles, the shared input data from the previous row has already been evicted from the L2 cache. + +For this optimization, you will remap the `blockIdx` variables using a custom tiling pattern (e.g., Z-curve, Morton ordering, or spatial clustering) to force the hardware to execute spatially adjacent output tiles simultaneously. Because these adjacent blocks share the same input data, executing them concurrently drastically increases L2 cache reuse. You must prove via Nsight Compute metrics that your reordering improves L2 cache hit rates and reduces global memory traffic. \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..7195696 --- /dev/null +++ b/Makefile @@ -0,0 +1,68 @@ +# Compiler and flags +NVCC = nvcc +CFLAGS = -O3 -arch=sm_86 -std=c++17 -rdc=true -g -lineinfo + +# Include and library paths for cuBLAS +CUBLAS_INCLUDES = -I/usr/local/cuda/include +CUBLAS_LIBS = -L/usr/local/cuda/lib64 -lcublas -lcublasLt + +# CUTLASS include path +CUTLASS_INCLUDES = -I/work/hdd/bche/hshah6/cutlass/include + +# Kernel files +KERNELS = $(wildcard kernels/*.cuh) +CPU_KERNELS = $(wildcard cpu_kernels/*.cuh) + +BUILD_DIR := build +SUFFIX := $(word 2, $(MAKECMDGOALS)) +BIN_SUFFIX := $(if $(SUFFIX),_$(SUFFIX),) + +# Default target +all: output_verification_rand test_gpt2 test_gpt2_kernels next_token_generation verify_kv_cache + +# Individual targets +.output_verification_rand: output_verification_rand +output_verification_rand: output_verification_rand.cu $(KERNELS) $(CPU_KERNELS) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) -c output_verification_rand.cu -o output_verification_rand.o + $(NVCC) $(CFLAGS) -o output_verification_rand output_verification_rand.o $(CUBLAS_LIBS) + +.test_gpt2: test_gpt2 +test_gpt2: $(BUILD_DIR)/test_gpt2$(BIN_SUFFIX) + +$(BUILD_DIR)/test_gpt2$(BIN_SUFFIX): test_gpt2.cu $(KERNELS) + @mkdir -p $(BUILD_DIR) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) \ + -c test_gpt2.cu -o $(BUILD_DIR)/test_gpt2.o + $(NVCC) $(CFLAGS) \ + -o $@ $(BUILD_DIR)/test_gpt2.o $(CUBLAS_LIBS) + +.test_gpt2_kernels: test_gpt2_kernels +test_gpt2_kernels: test_gpt2_kernels.cu $(KERNELS) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) -c test_gpt2_kernels.cu -o test_gpt2_kernels.o + $(NVCC) $(CFLAGS) -o test_gpt2_kernels test_gpt2_kernels.o $(CUBLAS_LIBS) + +.next_token_generation: next_token_generation +next_token_generation: next_token_generation.cu $(KERNELS) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) -c next_token_generation.cu -o next_token_generation.o + $(NVCC) $(CFLAGS) -o next_token_generation next_token_generation.o $(CUBLAS_LIBS) + +# for milestone 3 +.local_attn_verify: local_attn_verify +local_attn_verify: local_attn_verify.cu $(KERNELS) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) -g -c local_attn_verify.cu -o local_attn_verify.o + $(NVCC) $(CFLAGS) -g -o local_attn_verify local_attn_verify.o $(CUBLAS_LIBS) + +.verify_kv_cache: verify_kv_cache +verify_kv_cache: verify_kv_cache.cu $(KERNELS) + $(NVCC) $(CFLAGS) $(CUBLAS_INCLUDES) $(CUTLASS_INCLUDES) -c verify_kv_cache.cu -o verify_kv_cache.o + $(NVCC) $(CFLAGS) -o verify_kv_cache verify_kv_cache.o $(CUBLAS_LIBS) + +# Clean target +clean: + rm -rf $(BUILD_DIR) + rm -f *.o output_verification_rand test_gpt2 model_output next_token_generation local_attn_verify verify_kv_cache + +%: + @: + +.PHONY: all output_verification_rand model_output test_gpt2 test_gpt2_kernels next_token_generation local_attn_verify verify_kv_cache clean diff --git a/README.md b/README.md index 0640708..2f5f33f 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,217 @@ -# gpt-2-inference-engine +# ECE408 Final Project - GPT-2 +## Table of Contents + + - [GPT-2](#gpt-2) + - [Notes on Teamwork](#notes-on-teamwork) + - [Project Details](#project-details) + - [Project Overview](#project-overview) + - [Talk to the GPT2 You Wrote!](#talk-to-the-gpt2-you-wrote) + - [Milestone Deadlines](#milestone-deadlines) + - [Demos](#demos) + - [Grading Rubric](#grading-rubric) + - [Final Notes](#final-notes) + - [License and Academic Integrity Information](#license-and-academic-integrity-information) + +## GPT-2 + +[GPT-2](https://github.com/openai/gpt-2) (Generative Pretrained Transformer 2) is a transformer-based language model developed by OpenAI, released in 2019. GPT-2 is based on the Transformer architecture, which was first introduced in the paper [Attention is All You Need](https://arxiv.org/pdf/1706.03762). It is one of the first large-scale models to showcase the power of unsupervised learning for natural language generation tasks. GPT-2 is one of the works that marked the point of scaling law, where scale (i.e., model size, data size, and computational resources) was identified as a critical factor in performance. In addition, GPT-2 is part of a lineage of models that leverages transformer architecture, focusing on autoregressive generation, meaning it predicts the next word in a sequence given all the previous words using a decoder-only architecture. This paradigm has proved to be scalable and was adopted in later widely used works including ChatGPT. Therefore, hardware optimization and acceleration based on GPT-2 or other decoder-only transformers has become a crucial topic in parallel programming application. + +Because of the significance of GPT-2 and its relatively manageable model size, you are tasked with using CUDA to accelerate GPT-2's inference (forward pass) performance. + +### Industry Relevance +The traditional ECE 408 project has been the CNN project, which is still fundamental to computer vision. However, the industry has shifted aggressively towards transformers, which is the architecture behind LLMs such as ChatGPT, Claude, and Gemini. This project was created in part to bridge this gap. The concepts and technical skills (kernel optimization) you learn through this project have extremely high relevance to the current state of the industry. + +While the project may seem daunting in terms of the workload, with a capable team of 4 members, the workload is roughly comparable to the CNN Project. + +## Notes on Teamwork + +In this project, you and your teammates will work, present, and be graded as a team. This means that all members share responsibility for the final outcome and are expected to contribute meaningfully to the project. It is essential that you always actively communicate with each other about your individual progress, challenges, and plans for the project. This is especially important given the required milestone demo meetings, as every group member can affect your grade. As your grade is based on the work of other members of your team, please be fully confident in who you are choosing to work with. + +## Project Details + +This project is designed to be a team-based, multi-milestone project spanning the remainder of the semester. You need to form a team of 3-4 students (enrolled in the course) to work on this project. Each team will be assigned a course staff member as their mentor/grader for the project. We recommend having a team of 4 students if possible, as this project is quite large in scope, and the workload for each student is significant. Teams with fewer than 3 or more than 4 students will not be allowed. We recommend you use Campuswire or [Discord](https://discord.gg/X2GnMTXHEW) to find a team, however you can also sign up through the form on Campuswire without a team and we will attempt to match you. + +In this Project, you and your team will implement the entire forward pass (inference) of GPT-2 using CUDA, perform various optimizations, and thoroughly analyze the performance of your implementation using Nvidia profiling tools. Inference in this case means we will provide you a pre-trained model of GPT-2, and you implementation must solely generate new responses. In the last milestone, you will also have the chance to explore a few additional optimizations of your choice and further improve the performance of your implementation. + +**Note**: The Grace Period Policy for assignment submissions applies to the GPT Final Project as well. + +### GPT-2 Architecture +Below is a brief introduction to the overall architecture of GPT-2. If you already have a good understanding of the Transformer architecture, feel free to skim through the below sections or simply skip ahead to the other sections. Otherwise, we recommend gaining a clear understanding of the algorithms used in the GPT-2 architecture. Although we don't expect you to understand 100% of the details (ECE 408 is not an AI/ML course), you should understand the core concepts and goals of each component, as it will certainly help you with many aspects of this project. We also highly recommend [3Blue1Brown's Youtube Series on Transformers](https://www.youtube.com/watch?v=wjZofJX0v4M&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=6) for a more visual explanation of the Transformer architecture. + +#### Transformer Decoder Architecture +Similar to [Transformer-Decoder](https://arxiv.org/abs/1801.10198) as shown in Figure below , GPT-2 is composed of stacked transformer decoder layers. Each decoder block contains Multi-Head Self-Attention, Feed-Forward Network (FFN), Layer Normalization and Residual Connections. GPT-2's full model consists of 48 transformer layers (blocks) for the 1.5 billion parameter version. +![Alt text](assets/transformer-decoder.png "Transformer-Decoder") + +#### Multi-Head Self-Attention Mechanism +GPT-2 uses Multi-Head self-attention to attend to all previous tokens in the sequence to predict the next token. The model processes the input sequence and computes attention scores for each token, based on its relationships with the other tokens in the sequence. In multi-head self-attention, we perform the self-attention mechanism `h` times, where each head learns its own set of WQ(h), WK(h), and WV(h) weight matrices. + +For each head `h`, the attention is computed independently: + +head_h = softmax(Qh * Kh T / sqrt(dk)) * Vh + +Where: +- Qh = X * WQ(h) +- Kh = X * WK(h) +- Vh = X * WV(h) + +After computing attention for all heads, the outputs of each head are concatenated: + +MultiHead(Q, K, V) = Concat(head1, head2, ..., headh) * WO + +#### Causal Masking +GPT-2 uses causal masking in its self-attention mechanism. This ensures that when predicting the next token, the model only attends to the tokens that have come before it and does not look ahead. This masking enforces the autoregressive property, allowing GPT-2 to predict the next word in sequence rather than processing all tokens simultaneously. Pay attention to this in the softmax implementation. Note that the way we implement causal masking is different than the way presented in the 3Blue1Brown video. Take a look at `softmax_forward_cpu` inside of `cpu_kernels/attention.cuh`, and see if you can spot the difference. + +#### Feed-Forward Network (FFN) +After the self-attention operation, the output is passed through a feed-forward network, consisting of two linear transformations (Matrix Multiplication) with a GeLU activation in between. A GeLU activation is just a smooth version of ReLU that is differentiable (great for an activation function). It introduces non-linearity and can capture complex patterns. + +#### Layer Normalization and Residual Connections +Like other transformer-based models, GPT-2 uses residual connections and layer normalization to stabilize and improve the learning process. Like the name suggests, layer normalization normalizes across the feature dimension for each token. Furthermore, residual connections help propagate results of all layers to the final output. + +#### Encoder +GPT-2 uses a decoder-only architecture. It does not have a learnable encoder network model. But you might ask: Why is there an "encoder" kernel in this project? The answer is that here the "encoder" kernel is simply used to convert the input tokens into their corresponding embeddings, which are then fed into the transformer layers. + +## Project Overview + +To be able to pull code from the release repository, add it using the following command: + + git remote add gpt_release https://github.com/illinois-ece408/sp26_ece408_.gpt-release.git + + You can run `git remote -v` to verify that the repository was added. Then, you can fetch code for each milestone released using the following commands: + + git fetch gpt_release + + git merge gpt_release/main -m "funny comment" --allow-unrelated-histories + + If you see a merge conflict at this step, you may manually resolve the conflict or have Git accept all upstream changes. Make sure you have all contents of the milestone release (especially this README file) in your repository before proceeding. Then you may push the changes to your team's repository using: + + git push origin main + + + +For this project, you will only need to write and optimize the forward pass CUDA kernels in the `kernels` folder. + +For each kernel, you need to implement each of the functions marked with `// Implement this`. We have provided non-parallel reference implementations for each of the kernels using CPU in the `cpu_kernels` folder. These reference implementations do not make use of the GPU. Your job is to utilize your parallel programming knowledge to rewrite (and improve upon) the kernels so that they take full advantage of the parallelism available on GPUs. + +Below is a brief overview of the main kernels that you'll be working with in this project: + + - `encoder_forward`: Converts the model's input tokens into a compact, meaningful representation for the processing in the model's forward pass. + - `layernorm_forward`: Stabilizes the outputs of the model. This step is crucial for maintaining consistent behavior in the model. + - `matmul_forward`: Performs matrix multiplication to transform inputs using learned weights. + - `attention_forward`: Assigns dynamic weights to different parts of the input, allowing the model to focus on relevant information for better understanding. + - `residual_forward`: Helps to preserve information from previous layers while adding new information from subsequent layers. + - `gelu_forward`: Introduces non-linearity into the model, allowing it to capture more complex patterns. + + +**Note:** You are expected to adhere to University of Illinois academic integrity standards. Do not attempt to subvert any of the performance-measurement aspects of the final project. If you are unsure about whether something does not meet those guidelines, ask a member of the GPT Project course staff. + +## Talk to the GPT2 You Wrote! + +Once you have implemented the forward pass correctly, you will be able to talk* to the GPT-2 model you wrote! You can input text into the model and have it complete the text for you. To do this, run the command + + make next_token_generation + +Then, go into `generate_tokens.slurm` and edit the text you want the model to complete. After that, run + + sbatch generate_tokens.slurm + +Once the job has completed, you can then find the model outputs in `generate_tokens_output.out`! + +*\*talk: voice recognition not included, and no fancy chatbot capabilities, but by inputting text and having the model complete it, you can still technically talk to the GPT-2 model you wrote!* + + +## Milestone Deadlines + +All deliverables are due at **11:59 PM US Central Time** on the due date for each Milestone. + +| Milestone | Due | +| ----------- | ---------------------- | +| Milestone 1 | March 6th | +| Milestone 2 | April 10th | +| Milestone 3 | May 1st | + +**Note**: The 3-day grace period policy applies to the GPT Project milestones as well. However, the grace period does NOT apply to the demo dates. This means that you still have to sign up and attend the demo on the scheduled date even if you are using the grace period for that milestone for your other deliverables (e.g. code and/or report). + +## Demos + +As part of this project, all team members are required to attend a demo meeting after every milestone is due. At a demo, you are required to answer questions regarding each milestone to demonstrate your understanding of the milestone contents. Each member is responsible for all the topics covered in the milestone, and questions will be asked individually without help from other team members. We also reserve the right to ask additional questions regarding code that you have written, in which case the team can answer together. Each demo will last approximately 15-20 minutes. + +Demo questions are conceptual, which means you will need to understand why each kernel/optimization is written a specific way and how they all work together. The demo questions will involve little to no calculations. Make sure you are familiar with **all parts** of the milestone before the demo. Each team member's demo grade will depend on both the correctness of their own answer, as well as their teammates' answers. 70% of your demo grade will be the individual component, and 30% will be from the team component. **In other words, to receive full credit for a demo, every member needs to correctly answer their question.** *Partial credit will be given if you answered your demo question correctly, but one or more of your teammates did not.* (If you answered your question incorrectly, you will receive a zero for your demo grade in that milestone) + +Demos will always take place on the Monday after each milestone deadline. Each team will need to sign up for one demo slot using a Google Sheet, which we will release closer to the demo date. If any member(s) of the team cannot attend the demo on the scheduled date for any reason, please reach out to GPT Project course staff as soon as possible. + +### Demo Schedule and Location + +- Milestone 1 Demo: March 9th +- Milestone 2 Demo: April 13th +- Milestone 3 Demo: May 4th + +All demos are in-person at the **NCSA**. Please show up on time for the demo slot you signed up for and do not congregate and/or make loud conversations outside the room as there are NCSA staff offices around. + +*Also make sure to arrive early for your first demo, you may find it difficult to locate the demo room if you're not already familiar with the NCSA building layout.* + +## Grading Rubric + +1. Milestone 1 ( 15% ) + - Baseline Implementation Correctness ( 10% ) + - M1 Demo ( 5% ) +2. Milestone 2 ( 30% ) + - M2 Required Optimizations Correctness ( 18% ) + - Profiling Report and Optimization Proposal ( 7% ) + - M2 Demo ( 5% ) +3. Milestone 3 ( 50% ) + - M3 Required Optimizations Correctness ( 20% ) + - Additional Optimizations Code Correctness ( 10% ) + - Final Report ( 15% ) + - M3 Demo ( 5% ) +4. Subjective Evaluation ( 5% ) +5. Extra Credit ( capped up to 10% ) + - Optimizations Beyond M3 Requirements ( see [M3_README.md](M3_README.md) ) + - Competition Leaderboard Placement ( TBD ) + +**Note on Code Deliverables:** You and your teammates are fully responsible for ensuring that your code is ready and submitted on time. This means that any last-minute bugs in code submitted to the main branch will be counted against your team's grade, with compilation errors resulting in a 0 for your whole team's code grade in that milestone. + + +### Code Style/Clarity Point Deduction +Please note that as a student in a 400-level technical course, you are expected to write code that is clear and easy to read, or comment your code for any potentially confusing sections. Course staff reserve the right to deduct your team's Subjective Evaluation points if your code is deemed unusually difficult to read. + +We expect to *NOT* deduct code style/clarity points for most (hopefully all) groups in this project, but we want to make sure you are aware of this policy and that you are writing code with good variable names and comments. + +## Final Notes + +Please understand that this is a new project recently introduced to ECE 408. We are actively working on improving the project and your feedback is crucial for us to make this project better! We appreciate your initiative on participating on this new project, please do not hesitate to reach out to us if you have any questions or spot any errors in our code/instructions. + +While this project can seem overwhelming at times, we truly hope that you will find it rewarding and fun! + +We are excited to hear from you soon, good luck! + +## License and Academic Integrity Information + +You are **NOT** allowed to share any part of the ECE 408 GPT Final Project to anyone outside of your team. This includes, but is not limited to, sharing/publishing online the project description, code, and any other project-related materials. + +The ECE 408 GPT Final Project is based on [llm.c](https://github.com/karpathy/llm.c?tab=readme-ov-file#llmc), an open-source project led by ex-OpenAI research scientist Andrej Karpathy. The original project is licensed under the MIT License. + +Although you are free to explore llm.c (and we encourage you to do so if you would like to benchmark your work against it), please note that you are **NOT** allowed to copy any part of the llm.c project into your submission without proper citation and permission from the course staff. + +As this project is part of the coursework at UIUC, it is essential to adhere to the University's academic integrity policies. Please refrain from using any external sources, repositories, or unauthorized online materials to complete your work. + +AI usage should be limited for your own benefit, as modern LLMs often struggle with the specifics of parallel programming and CUDA, and tends to introduce nasty bugs. It is ok to use AI to assist with writing comments, improving code readability, and other non-core programming tasks. However, any non-trivial usage of AI tools must be properly cited, and you must be prepared to answer questions regarding your code. + +Violations of academic integrity, including plagiarism or unauthorized collaboration with members outside of your team, will be taken seriously and may result in disciplinary actions, including but not limited to a grade of zero for the project and reporting to the University. + +Please refer to the [University of Illinois Academic Integrity Policy](https://studentcode.illinois.edu/) for more details. + + +### More Information on Software Licenses + +Software licenses play an important role in protecting intellectual property while enabling collaboration in the software development community. It is essential to understand the implications of different licenses when it comes to sharing, modifying, and distributing software. + +#### Some common types of software licenses include: + +**MIT License**: One of the most permissive licenses, requiring only attribution while allowing commercial use. It's particularly popular in educational projects and small open-source tools. + +**GNU GPL License**: Takes a stronger stance on software freedom. It enforces "copyleft" principles, requiring that any derivative works must also be open source. This license has been fundamental to the growth of Linux and many other major open-source projects. + +**Apache License 2.0**: Provides a middle ground, offering similar permissions to MIT while also including explicit patent rights. This makes it particularly suitable for corporate use and large-scale projects. + +Authors can also choose to reserve all rights to their software, making it proprietary and closed-source. \ No newline at end of file diff --git a/Temperature1.5_Final_Report.pdf b/Temperature1.5_Final_Report.pdf new file mode 100644 index 0000000..582802d Binary files /dev/null and b/Temperature1.5_Final_Report.pdf differ diff --git a/assets/transformer-decoder.png b/assets/transformer-decoder.png new file mode 100644 index 0000000..714a047 Binary files /dev/null and b/assets/transformer-decoder.png differ diff --git a/build/test_gpt2_op_11 b/build/test_gpt2_op_11 new file mode 100755 index 0000000..fe73dbf Binary files /dev/null and b/build/test_gpt2_op_11 differ diff --git a/build/test_gpt2_op_12 b/build/test_gpt2_op_12 new file mode 100755 index 0000000..a096003 Binary files /dev/null and b/build/test_gpt2_op_12 differ diff --git a/build/test_gpt2_op_8 b/build/test_gpt2_op_8 new file mode 100755 index 0000000..1003be1 Binary files /dev/null and b/build/test_gpt2_op_8 differ diff --git a/build/test_gpt2_op_9 b/build/test_gpt2_op_9 new file mode 100755 index 0000000..b4550fb Binary files /dev/null and b/build/test_gpt2_op_9 differ diff --git a/build/test_gpt2_req_4 b/build/test_gpt2_req_4 new file mode 100755 index 0000000..9511491 Binary files /dev/null and b/build/test_gpt2_req_4 differ diff --git a/cpu_kernels/attention.cuh b/cpu_kernels/attention.cuh new file mode 100644 index 0000000..7e77c16 --- /dev/null +++ b/cpu_kernels/attention.cuh @@ -0,0 +1,139 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef ATTENTION_CUH +#define ATTENTION_CUH + +#include +#include +#include + +void permute_kernel_cpu(float* q, float* k, float* v, + const float* inp, + int B, int T, int NH, int d) { + for (int b = 0; b < B; ++b) { + for (int nh_idx = 0; nh_idx < NH; ++nh_idx) { + for (int n = 0; n < T; ++n) { + for (int d_ = 0; d_ < d; ++d_) { + int inp_idx = (b * T * 3 * NH * d) + (n * 3 * NH * d) + (0 * NH * d) + (nh_idx * d) + d_; + q[b * NH * T * d + nh_idx * T * d + n * d + d_] = inp[inp_idx]; + k[b * NH * T * d + nh_idx * T * d + n * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * T * d + nh_idx * T * d + n * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } + } + } + } +} + +void unpermute_kernel_cpu(float* inp, float* out, int B, int T, int NH, int d) { + for (int b = 0; b < B; ++b) { + for (int n = 0; n < T; ++n) { + for (int nh_idx = 0; nh_idx < NH; ++nh_idx) { + for (int d_ = 0; d_ < d; ++d_) { + int other_idx = (b * NH * T * d) + (n * NH * d) + (nh_idx * d) + d_; + out[other_idx] = inp[b * NH * T * d + nh_idx * T * d + n * d + d_]; + } + } + } + } +} + + +void softmax_forward_cpu(float* out, float inv_temperature, const float* inp, int N, int T) { + for (int idx = 0; idx < N * T; ++idx) { + // The thread's position in its current row + int own_pos = idx % T; + + // Pointer to the current row + const float* x = inp + idx * T; + + float maxval = -FLT_MAX; + // Think about what this for loop condition is doing (Note the significance of i <= own_pos) + for (int i = 0; i <= own_pos; ++i) { + maxval = fmaxf(maxval, x[i]); + } + + // Compute softmax + float sumval = 0.0f; + for (int i = 0; i <= own_pos; ++i) { + float ev = expf(inv_temperature * (x[i] - maxval)); + sumval += ev; + out[idx * T + i] = ev; + } + + // Normalize + float norm = 1.0f / sumval; + for (int i = 0; i <= own_pos; ++i) { + out[idx * T + i] *= norm; + } + } +} + +void attention_forward_cpu(float* out, float* qkvr, float* att, + float* inp, + int B, int T, int C, int NH) { + // Note: `inp` is re-used as a scratch buffer. + // Its contents will be overwritten by this function. + + // inp is (B, T, 3C) QKV + // preatt, att are (B, NH, T, T) + // output is (B, T, C) + int HS = C / NH; // head size + + // permute and separate inp from (B, T, 3, NH, HS) to 3X (B, NH, T, HS) + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + permute_kernel_cpu(q, k, v, inp, B, T, NH, HS); + + // Attention matmul: Q @ K^T + // Compute pre-attention scores (B, NH, T, T) + float* preatt = inp; + for (int b = 0; b < B; b++) { + for (int nh = 0; nh < NH; nh++) { + for (int t1 = 0; t1 < T; t1++) { + for (int t2 = 0; t2 < T; t2++) { + float sum = 0.0f; + for (int hs = 0; hs < HS; hs++) { + sum += k[b * NH * T * HS + nh * T * HS + t2 * HS + hs] * + q[b * NH * T * HS + nh * T * HS + t1 * HS + hs]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } + } + } + } + + // Compute the softmax + float scale = 1.0 / sqrtf(HS); + softmax_forward_cpu(att, scale, preatt, B * NH, T); + + + float* vaccum = inp; + // Attention matmul: P @ V, where P holds the attention probabilities + // (B, NH, T, T) @ (B, NH, T, HS) -> (B, NH, T, HS) + for (int b = 0; b < B; ++b) { + for (int nh = 0; nh < NH; ++nh) { + for (int t = 0; t < T; ++t) { + for (int hs = 0; hs < HS; ++hs) { + float sum = 0.0f; + for (int t2 = 0; t2 < T; ++t2) { + sum += att[b * NH * T * T + nh * T * T + t * T + t2] * + v[b * NH * T * HS + nh * T * HS + t2 * HS + hs]; + } + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } + } + } + } + + unpermute_kernel_cpu(vaccum, out, B, T, NH, HS); +} + +#endif // ATTENTION_CUH \ No newline at end of file diff --git a/cpu_kernels/encoder.cuh b/cpu_kernels/encoder.cuh new file mode 100644 index 0000000..527fd8f --- /dev/null +++ b/cpu_kernels/encoder.cuh @@ -0,0 +1,27 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef ENCODER_CUH +#define ENCODER_CUH + +#include +#include +#include + +void encoder_forward_cpu(float* out, const int* inp, const float* wte, const float* wpe, + int B, int T, int C) { + for (int b = 0; b < B; ++b) { + for (int token_pos = 0; token_pos < T; ++token_pos) { + for (int c = 0; c < C; ++c) { + int token = inp[b * T + token_pos]; + out[b * T * C + token_pos * C + c] = wte[token * C + c] + wpe[token_pos * C + c]; + } + } + } +} + +#endif // ENCODER_CUH \ No newline at end of file diff --git a/cpu_kernels/gelu.cuh b/cpu_kernels/gelu.cuh new file mode 100644 index 0000000..ee25430 --- /dev/null +++ b/cpu_kernels/gelu.cuh @@ -0,0 +1,26 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef GELU_CUH +#define GELU_CUH + +#include +#include +#include + +#define GELU_SCALING_FACTOR sqrtf(2.0f / M_PI) + +void gelu_forward_cpu(float* out, const float* inp, int N) { + for (int i = 0; i < N; ++i) { + float val = inp[i]; + float cube = 0.044715f * val * val * val; + // Think about what GELU is, are we computing the exact gelu function here? + out[i] = 0.5f * val * (1.0f + tanhf(GELU_SCALING_FACTOR * (val + cube))); + } +} + +#endif // GELU_CUH \ No newline at end of file diff --git a/cpu_kernels/layernorm.cuh b/cpu_kernels/layernorm.cuh new file mode 100644 index 0000000..b29efcd --- /dev/null +++ b/cpu_kernels/layernorm.cuh @@ -0,0 +1,43 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef LAYERNORM_CUH +#define LAYERNORM_CUH + +#include +#include +#include + +void layernorm_forward_cpu(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + for (int b = 0; b < B; ++b) { + for (int t = 0; t < T; ++t) { + // For each token in the batch, normalize across the channel dimension + float sum = 0.0f; + for (int c = 0; c < C; ++c) { + sum += inp[b * T * C + t * C + c]; + } + float m = sum / C; + + sum = 0.0f; + for (int c = 0; c < C; ++c) { + float diff = inp[b * T * C + t * C + c] - m; + sum += diff * diff; + } + // Standard deviation of the channel values + // but think about why we take the reciprocal + float s = rsqrtf(sum / C + 1e-5f); + + for (int c = 0; c < C; ++c) { + float n = s * (inp[b * T * C + t * C + c] - m); + out[b * T * C + t * C + c] = n * weight[c] + bias[c]; + } + } + } +} + +#endif // LAYERNORM_CUH \ No newline at end of file diff --git a/cpu_kernels/local_attention.cuh b/cpu_kernels/local_attention.cuh new file mode 100644 index 0000000..d7b5e9f --- /dev/null +++ b/cpu_kernels/local_attention.cuh @@ -0,0 +1,76 @@ +#ifndef LOCAL_ATTENTION_CUH +#define LOCAL_ATTENTION_CUH + +#include +#include +#include +#include "attention.cuh" + +#define WINDOW_SIZE 128 + +void local_attention_forward_cpu(float* out, float* qkvr, float* inp, + int B, int T, int NH, int HS) { + + float *q, *k, *v; + q = qkvr + 0 * B * T * NH * HS; + k = qkvr + 1 * B * T * NH * HS; + v = qkvr + 2 * B * T * NH * HS; + + // permute and separate inp from (B, T, 3, NH, HS) to 3X (B, NH, T, HS) + permute_kernel_cpu(q, k, v, inp, B, T, NH, HS); + + float scale = 1.0 / sqrtf(HS); + float* preatt = inp; + for(int b = 0; b < B; b++){ + for(int nh = 0; nh < NH; nh++){ + for(int q_t = 0; q_t < T; q_t++){ + // weights buffer for the current window + float* weights = (float*) calloc(WINDOW_SIZE, sizeof(float)); + + // causal window start for keys + int k_start = max(0, q_t - WINDOW_SIZE); + + // raw attention scores within the window + for(int k_t = k_start; k_t <= q_t; k_t++){ + float sum = 0.0f; + for(int hs = 0; hs < HS; hs++){ + sum += k[b * NH * T * HS + nh * T * HS + k_t * HS + hs] * + q[b * NH * T * HS + nh * T * HS + q_t * HS + hs]; + } + weights[q_t - k_t] = sum; + } + + // softmax over the window + float maxval = -FLT_MAX; + for(int k_t = k_start; k_t <= q_t; k_t++){ + maxval = fmaxf(maxval, weights[q_t - k_t]); + } + float sumval = 0.0f; + for(int k_t = k_start; k_t <= q_t; k_t++){ + float ev = expf(scale * (weights[q_t - k_t] - maxval)); + sumval += ev; + weights[q_t - k_t] = ev; + } + + for(int k_t = k_start; k_t <= q_t; k_t++){ + weights[q_t - k_t] /= sumval; + } + + // weighted sum of values + for(int hs = 0; hs < HS; hs++){ + float sum = 0; + for(int k_t = k_start; k_t <= q_t; k_t++){ + sum += weights[q_t - k_t] * v[b * NH * T * HS + nh * T * HS + k_t * HS + hs]; + } + preatt[b * NH * T * HS + nh * T * HS + q_t * HS + hs] = sum; + } + + free(weights); + } + } + } + + unpermute_kernel_cpu(preatt, out, B, T, NH, HS); +} + +#endif // LOCAL_ATTENTION_CUH \ No newline at end of file diff --git a/cpu_kernels/matmul.cuh b/cpu_kernels/matmul.cuh new file mode 100644 index 0000000..9427804 --- /dev/null +++ b/cpu_kernels/matmul.cuh @@ -0,0 +1,35 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef MATMUL_CUH +#define MATMUL_CUH + +#include +#include +#include + +void matmul_forward_cpu(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Input (B,T,C) @ Weight (OC, C) + Bias is (OC) = Output (B,T,OC) + for (int b = 0; b < B; b++) { + for (int t = 0; t < T; t++) { + float* out_token_ptr = out + b * T * OC + t * OC; + const float* inp_token_ptr = inp + b * T * C + t * C; + for (int o = 0; o < OC; o++) { + float acc = (bias != NULL) ? bias[o] : 0.0f; + const float* weight_row_ptr = weight + o*C; + // Dot product of the input token values and corresponding row of weight values + for (int i = 0; i < C; i++) { + acc += inp_token_ptr[i] * weight_row_ptr[i]; + } + out_token_ptr[o] = acc; + } + } + } +} + +#endif // MATMUL_CUH \ No newline at end of file diff --git a/cpu_kernels/residual.cuh b/cpu_kernels/residual.cuh new file mode 100644 index 0000000..a0f7988 --- /dev/null +++ b/cpu_kernels/residual.cuh @@ -0,0 +1,22 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef RESIDUAL_CUH +#define RESIDUAL_CUH + +#include +#include +#include + +void residual_forward_cpu(float* out, const float* inp1, const float* inp2, int N) { + // Element-wise addition for residual connections + for (int i = 0; i < N; ++i) { + out[i] = inp1[i] + inp2[i]; + } +} + +#endif // RESIDUAL_CUH \ No newline at end of file diff --git a/generate_tokens.slurm b/generate_tokens.slurm new file mode 100644 index 0000000..e5a7dae --- /dev/null +++ b/generate_tokens.slurm @@ -0,0 +1,21 @@ +#!/bin/bash +#SBATCH --job-name="generate_tokens" +#SBATCH --output="generate_tokens_output.out" +#SBATCH --error="generate_tokens_errors.err" +#SBATCH --partition=gpuA40x4-interactive +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="scratch" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:25:00 + +module reset + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +# run model +srun ./next_token_generation "The meaning of life is" diff --git a/gpt2.cuh b/gpt2.cuh new file mode 100644 index 0000000..abfba8c --- /dev/null +++ b/gpt2.cuh @@ -0,0 +1,413 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include +#include +#include +#include +#include +#include +#include +#include + +// GPU / CUDA related +#include +#include +#include "utils/utils.h" +#include "utils/tokenizer.h" +#include "utils/cuda_utils.cuh" + +// kernels +#include "kernels/attention.cuh" +#include "kernels/encoder.cuh" +#include "kernels/gelu.cuh" +#include "kernels/layernorm.cuh" +#include "kernels/matmul.cuh" +#include "kernels/residual.cuh" + +// ---------------------------------------------------------------------------- +// GPT-2 model definition + +typedef struct { + int max_seq_len; + int vocab_size; + int padded_vocab_size; // padded to multiple of 128 + int num_layers; + int num_heads; + int channels; +} GPT2Config; + +// the parameters of the model +#define NUM_PARAMETER_TENSORS 16 +typedef struct { + float* wte; // (Vp, C) + float* wpe; // (maxT, C) + float* ln1w; // (L, C) + float* ln1b; // (L, C) + float* qkvw; // (L, 3*C, C) + float* qkvb; // (L, 3*C) + float* attprojw; // (L, C, C) + float* attprojb; // (L, C) + float* ln2w; // (L, C) + float* ln2b; // (L, C) + float* fcw; // (L, 4*C, C) + float* fcb; // (L, 4*C) + float* fcprojw; // (L, C, 4*C) + float* fcprojb; // (L, C) + float* lnfw; // (C) + float* lnfb; // (C) +} ParameterTensors; + +void fill_in_parameter_sizes(size_t* param_sizes, GPT2Config config) { + int Vp = config.padded_vocab_size; + int C = config.channels; + int maxT = config.max_seq_len; + int L = config.num_layers; + param_sizes[0] = Vp * C; // wte + param_sizes[1] = maxT * C; // wpe + param_sizes[2] = L * C; // ln1w + param_sizes[3] = L * C; // ln1b + param_sizes[4] = L * (3 * C) * C; // qkvw + param_sizes[5] = L * (3 * C); // qkvb + param_sizes[6] = L * C * C; // attprojw + param_sizes[7] = L * C; // attprojb + param_sizes[8] = L * C; // ln2w + param_sizes[9] = L * C; // ln2b + param_sizes[10] = L * (4 * C) * C; // fcw + param_sizes[11] = L * (4 * C); // fcb + param_sizes[12] = L * C * (4 * C); // fcprojw + param_sizes[13] = L * C; // fcprojb + param_sizes[14] = C; // lnfw + param_sizes[15] = C; // lnfb +} + +// allocate memory for the parameters and point the individual tensors to the right places +float* malloc_and_point_parameters(ParameterTensors* params, size_t* param_sizes, int on_device) { + // on_device: 0 = CPU, 1 = GPU + // calculate the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += param_sizes[i]; + } + // malloc all parameters all at once on the device + float* params_memory; + if (on_device) { + cudaCheck(cudaMalloc((void**)¶ms_memory, num_parameters * sizeof(float))); + } else { + params_memory = (float*)mallocCheck(num_parameters * sizeof(float)); + } + // assign all the tensors their place in the array + float** ptrs[] = { + ¶ms->wte, ¶ms->wpe, ¶ms->ln1w, ¶ms->ln1b, ¶ms->qkvw, ¶ms->qkvb, + ¶ms->attprojw, ¶ms->attprojb, ¶ms->ln2w, ¶ms->ln2b, ¶ms->fcw, ¶ms->fcb, + ¶ms->fcprojw, ¶ms->fcprojb, ¶ms->lnfw, ¶ms->lnfb + }; + float* params_memory_iterator = params_memory; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + *(ptrs[i]) = params_memory_iterator; + params_memory_iterator += param_sizes[i]; + } + return params_memory; +} + +#define NUM_ACTIVATION_TENSORS 21 +typedef struct { + float* encoded; // (B, T, C) + float* ln1; // (L, B, T, C) + float* ln1_mean; // (L, B, T) + float* ln1_rstd; // (L, B, T) + float* atty; // (L, B, T, C) + float* att; // (L, B, NH, T, T) + float* attproj; // (L, B, T, C) + float* residual2; // (L, B, T, C) + float* ln2; // (L, B, T, C) + float* ln2_mean; // (L, B, T) + float* ln2_rstd; // (L, B, T) + float* fch; // (L, B, T, 4*C) + float* fch_gelu; // (L, B, T, 4*C) + float* fcproj; // (L, B, T, C) + float* residual3; // (L, B, T, C) + float* lnf; // (B, T, C) + float* lnf_mean; // (B, T) + float* lnf_rstd; // (B, T) + float* losses; // (B, T) + float* qkvr; // (L, B, T, 3*C) + // This buffer will store the logits + // during the processing of transformer blocks, we will also use this as a + // general scratchpad buffer. Allocation is made large enough to hold (B, T, 3C), + // (B, NH, T, T), and (B, T, V) shaped tensors. + float* output; +} ActivationTensors; + +void fill_in_activation_sizes(size_t* act_sizes, int B, int T, GPT2Config config) { + size_t Vp = config.padded_vocab_size; + size_t L = config.num_layers; + size_t NH = config.num_heads; + size_t C = config.channels; + act_sizes[0] = B * T * C; // encoded + act_sizes[1] = L * B * T * C; // ln1 + act_sizes[2] = 1; // ln1_mean, reserved for future use + act_sizes[3] = 1; // ln1_rstd, reserved for future use + act_sizes[4] = L * B * T * C; // atty + act_sizes[5] = L * B * NH * T * T; // att + act_sizes[6] = L * B * T * C; // attproj + act_sizes[7] = L * B * T * C; // residual2 + act_sizes[8] = L * B * T * C; // ln2 + act_sizes[9] = 1; // ln2_mean, reserved for future use + act_sizes[10] = 1; // ln2_rstd, reserved for future use + act_sizes[11] = L * B * T * 4*C; // fch + act_sizes[12] = L * B * T * 4*C; // fch_gelu + act_sizes[13] = L * B * T * C; // fcproj + act_sizes[14] = L * B * T * C; // residual3 + act_sizes[15] = B * T * C; // lnf + act_sizes[16] = 1; // lnf_mean, reserved for future use + act_sizes[17] = 1; // lnf_rstd, reserved for future use + act_sizes[18] = B * T; // losses + act_sizes[19] = L * B * T * 3*C; // qkvr + act_sizes[20] = B * T * max(3*C, max(NH*T, Vp)); // output / scratch +} + +// Reserved for future use +#define NUM_BACKWARD_TENSORS 3 +typedef struct { + float* bt4c; // (B, T, 4*C) + float* preatt; // (B, NH, T, T) + float* residual3; // (B, T, C) +} GradActTensors; + +float* malloc_and_point(float** targets[], const size_t* act_sizes, int n) { + size_t num_activations = 0; + for (size_t i = 0; i < n; i++) { + num_activations += act_sizes[i]; + } + float* acts_memory; + cudaCheck(cudaMalloc((void**)&acts_memory, num_activations * sizeof(float))); + // clear the allocated memory since cudaMalloc does not guarantee zeroed memory + cudaCheck(cudaMemset(acts_memory, 0, num_activations * sizeof(float))); + float* acts_memory_iterator = acts_memory; + for (size_t i = 0; i < n; i++) { + *(targets[i]) = acts_memory_iterator; + acts_memory_iterator += act_sizes[i]; + } + return acts_memory; +} + +float* malloc_and_point_activations(ActivationTensors* acts, const size_t* act_sizes) { + float** ptrs[] = { + &acts->encoded, &acts->ln1, &acts->ln1_mean, &acts->ln1_rstd, &acts->atty, + &acts->att, &acts->attproj, &acts->residual2, &acts->ln2, &acts->ln2_mean, + &acts->ln2_rstd, &acts->fch, &acts->fch_gelu, &acts->fcproj, &acts->residual3, &acts->lnf, + &acts->lnf_mean, &acts->lnf_rstd, &acts->losses, &acts->qkvr, &acts->output + }; + return malloc_and_point(ptrs, act_sizes, NUM_ACTIVATION_TENSORS); +} + +typedef struct { + GPT2Config config; + // the weights of the model, and their sizes + ParameterTensors params; + size_t param_sizes[NUM_PARAMETER_TENSORS]; + float* params_memory; + size_t num_parameters; + ParameterTensors grads; + float* grads_memory; + float* m_memory; + float* v_memory; + // the activations of the model, and their sizes + ActivationTensors acts; + size_t act_sizes[NUM_ACTIVATION_TENSORS]; + float* acts_memory; + size_t num_activations; + GradActTensors grads_acts; + size_t num_grad_acts; + float* grads_acts_memory; + // other run state configuration + int batch_size; // the batch size (B) of current forward pass + int seq_len; // the sequence length (T) of current forward pass + int* inputs; // the input tokens for the current forward pass + int* targets; // the target tokens for the current forward pass + float mean_loss; + float* cpu_losses; +} GPT2; + +void gpt2_build_from_checkpoint(GPT2 *model, const char* checkpoint_path) { + + // read in model from a checkpoint file + FILE *model_file = fopenCheck(checkpoint_path, "rb"); + int model_header[256]; + freadCheck(model_header, sizeof(int), 256, model_file); + if (model_header[0] != 20240326) { fprintf(stderr, "GPT2 - Bad magic model file\n"); exit(EXIT_FAILURE); } + if (model_header[1] != 3) { + // was bumped from 1 -> 3 to incorporate the padded vocab size + fprintf(stderr, "GPT2 - Bad version in model file\n"); + exit(EXIT_FAILURE); + } + + // read in hyperparameters + model->config.max_seq_len = model_header[2]; + model->config.vocab_size = model_header[3]; + model->config.num_layers = model_header[4]; + model->config.num_heads = model_header[5]; + model->config.channels = model_header[6]; + model->config.padded_vocab_size = model_header[7]; + + // allocate space for all the parameters and read them in + fill_in_parameter_sizes(model->param_sizes, model->config); + + // count the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += model->param_sizes[i]; + } + model->num_parameters = num_parameters; + + // create memory for model parameters on the device + model->params_memory = malloc_and_point_parameters(&model->params, model->param_sizes, 1); + + // read in all the parameters from file and copy them to device + float* params_memory_cpu = (float*)mallocCheck(num_parameters * sizeof(float)); + freadCheck(params_memory_cpu, sizeof(float), num_parameters, model_file); + cudaCheck(cudaMemcpy(model->params_memory, params_memory_cpu, num_parameters * sizeof(float), cudaMemcpyHostToDevice)); + free(params_memory_cpu); + fcloseCheck(model_file); + + // other inits + model->acts_memory = NULL; + model->grads_memory = NULL; + model->m_memory = NULL; + model->v_memory = NULL; + model->grads_acts_memory = NULL; + model->inputs = NULL; + model->targets = NULL; + model->cpu_losses = NULL; + model->batch_size = 0; + model->seq_len = 0; + model->mean_loss = -1.0f; // -1.0f will designate no loss +} + +void gpt2_forward(GPT2 *model, int* inputs, int B, int T) { + + // ensure the model was initialized or error out + if (model->params_memory == NULL) { + printf("GPT2 - Error: model was not initialized properly.\n"); + exit(EXIT_FAILURE); + } + + // convenience parameters + int V = model->config.vocab_size; + int Vp = model->config.padded_vocab_size; + int L = model->config.num_layers; + int NH = model->config.num_heads; + int C = model->config.channels; + + // validate inputs, all indices must be in the range [0, V) + for(int i = 0; i < B * T; i++) { + assert(0 <= inputs[i] && inputs[i] < V); + } + + // allocate space for all the activations if needed (done here, lazily) + if(model->acts_memory == NULL || B != model->batch_size || T != model->seq_len) { + // record the current B,T as well + model->batch_size = B; + model->seq_len = T; + // and now allocate the space + fill_in_activation_sizes(model->act_sizes, B, T, model->config); + size_t num_activations = 0; + for (size_t i = 0; i < NUM_ACTIVATION_TENSORS; i++) { + num_activations += model->act_sizes[i]; + } + model->num_activations = num_activations; + if (model->acts_memory != NULL) { + printf("GPT2 - Reallocating activations for B=%d T=%d\n", B, T); + cudaFree(model->acts_memory); + cudaFree(model->inputs); + } + model->acts_memory = malloc_and_point_activations(&model->acts, model->act_sizes); + printf("GPT2 - Allocated %zu MiB for activations\n", (num_activations * sizeof(float)) >> 20); // >> 20 is /(1024*1024) + cudaCheck(cudaMalloc((void**)&model->inputs, B * T * sizeof(int))); + cudaCheck(cudaMallocHost((void**)&model->cpu_losses, B * T * sizeof(float))); + } + + // copy inputs to the model + cudaCheck(cudaMemcpy(model->inputs, inputs, B * T * sizeof(int), cudaMemcpyHostToDevice)); + + // forward pass + ParameterTensors params = model->params; // for brevity + ActivationTensors acts = model->acts; + float* residual; + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe, B, T, C); // encoding goes into residual[0] + + for (int l = 0; l < L; l++) { + + residual = l == 0 ? acts.encoded : acts.residual3 + (l-1) * B * T * C; + + // get the pointers of the weights for this layer + float* l_ln1w = params.ln1w + l * C; + float* l_ln1b = params.ln1b + l * C; + float* l_qkvw = params.qkvw + l * 3*C * C; + float* l_qkvb = params.qkvb + l * 3*C; + float* l_attprojw = params.attprojw + l * C * C; + float* l_attprojb = params.attprojb + l * C; + float* l_ln2w = params.ln2w + l * C; + float* l_ln2b = params.ln2b + l * C; + float* l_fcw = params.fcw + l * 4*C * C; + float* l_fcb = params.fcb + l * 4*C; + float* l_fcprojw = params.fcprojw + l * C * 4*C; + float* l_fcprojb = params.fcprojb + l * C; + + // get the pointers of the activations for this layer + float* l_ln1 = acts.ln1 + l * B * T * C; + float* l_ln1_mean = acts.ln1_mean; + float* l_ln1_rstd = acts.ln1_rstd; + float* l_qkvr = acts.qkvr + l * B * T * 3*C; + float* l_atty = acts.atty + l * B * T * C; + float* l_att = acts.att + l * B * NH * T * T; + float* l_attproj = acts.attproj + l * B * T * C; + float* l_residual2 = acts.residual2 + l * B * T * C; + float* l_ln2 = acts.ln2 + l * B * T * C; + float* l_ln2_mean = acts.ln2_mean; + float* l_ln2_rstd = acts.ln2_rstd; + float* l_fch = acts.fch + l * B * T * 4*C; + float* l_fch_gelu = acts.fch_gelu + l * B * T * 4*C; + float* l_fcproj = acts.fcproj + l * B * T * C; + float* l_residual3 = acts.residual3 + l * B * T * C; + // these are only needed as scratchpads for the forward pass + float* scratch = acts.output; + + // now do the forward pass + layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C); + matmul_forward(scratch, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C); + attention_forward(l_atty, l_qkvr, l_att, scratch, B, T, C, NH); + matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C); + residual_forward(l_residual2, residual, l_attproj, B*T*C); + layernorm_forward(l_ln2, l_ln2_mean, l_ln2_rstd, l_residual2, l_ln2w, l_ln2b, B, T, C); + matmul_forward(l_fch, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + gelu_forward(l_fch_gelu, l_fch, B*T*4*C); + matmul_forward(l_fcproj, l_fch_gelu, l_fcprojw, l_fcprojb, B, T, 4*C, C); + residual_forward(l_residual3, l_residual2, l_fcproj, B*T*C); + } + + residual = acts.residual3 + (L-1) * B * T * C; // last residual is in residual3 + layernorm_forward(acts.lnf, acts.lnf_mean, acts.lnf_rstd, residual, params.lnfw, params.lnfb, B, T, C); + matmul_forward(acts.output, acts.lnf, params.wte, NULL, B, T, C, Vp); + + model->mean_loss = -1.0f; +} + +void gpt2_free(GPT2 *model) { + cudaCheck(cudaFree(model->params_memory)); + cudaCheck(cudaFree(model->grads_memory)); + cudaCheck(cudaFree(model->m_memory)); + cudaCheck(cudaFree(model->v_memory)); + cudaCheck(cudaFree(model->acts_memory)); + cudaCheck(cudaFree(model->grads_acts_memory)); + cudaCheck(cudaFree(model->inputs)); + cudaCheck(cudaFree(model->targets)); + cudaFreeHost(model->cpu_losses); +} + +#define GPT2_EOT 50256 \ No newline at end of file diff --git a/kernels/attention.cuh b/kernels/attention.cuh new file mode 100644 index 0000000..13df1f9 --- /dev/null +++ b/kernels/attention.cuh @@ -0,0 +1,105 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B* T* NH* T){ + int b = (x / T / T / NH); + int nh = (x / T / T) % NH; + int t1 = (x / T) % T; + int t2 = x % T; + + float sum = 0.0f; + for (int i = 0; i < HS; i++) { + sum += q[b*NH*T*HS + nh*T*HS + t1*HS + i] * k[b*NH*T*HS + nh*T*HS + t2*HS + i]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } +} + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + float sum = 0.0f; + for (int t2 = 0; t2 < T; t2++) { + sum += att[b*NH*T*T + nh*T*T + t*T + t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; + } + + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x-1) / block.x,1,1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels/attention_kv_cache_template.cuh b/kernels/attention_kv_cache_template.cuh new file mode 100644 index 0000000..eb9c490 --- /dev/null +++ b/kernels/attention_kv_cache_template.cuh @@ -0,0 +1,86 @@ +#ifndef __ATTENTION_KV_CACHE_TEMPLATE_CUH__ +#define __ATTENTION_KV_CACHE_TEMPLATE_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +extern bool g_enable_kv_cache; // Toggles KV-caching logic +extern float* g_kv_cache; // Pointer to the GPU-allocated KV cache buffer +extern int g_layer_idx; // Tracks the current transformer layer, useful for knowing when to switch to decode +extern bool g_is_prefill; // True during the initial prompt, False during token generation +extern int g_current_pos; // Tracks the absolute sequence position during Decode, helps index into the KV cache + +// --- Suggested Kernel Placeholders (TODO: Add your regular attention kernel here, and any other kernels you may need) --- + +/** + * Kernel to store the newly computed Key and Value into the global KV cache. + * CAN BE FUSED WITH QKV MATMUL + * In Prefill: Stores T tokens for the current layer. + * In Decode: Stores 1 token at g_current_pos for the current layer. + */ +__global__ void store_kv_kernel(/* args */) { + // Implement logic to copy K and V from the input QKV buffer into g_kv_cache +} + +/** + * Kernel to compute the attention for the Decode phase (T=1). + * This involves: + * 1. Extracting the current Query (Q) + * 2. Gathering all previous Keys (K) and Values (V) from the cache + * 3. Computing attention using the new Q and cached KV values. + */ +__global__ void decode_attention_kernel(/* args */) { + // Implement T=1 attention logic using the KV cache +} + +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // 1. Check if KV-caching is enabled, used by the verification script + if (!g_enable_kv_cache) { + // TODO: Call your standard Baseline Attention here (Milestone 1/2) + return; + } + + // 2. Initialize KV Cache if necessary + if (g_kv_cache == nullptr) { + // TODO: Allocate the KV cache, based accordingly to the model parameters + // Option 1: Allocate a very big kv cache to store all the tokens + // Option 2: Dynamically resize the kv cache repeatedly for each new token + } + + if (g_is_prefill) { + // --- PREFILL PHASE --- + // During prefill, we process the entire input sequence (T > 1). + + // A. Store the K and V results from this layer into the cache. + // The projections are available in 'inp' (the QKV matmul output). + + // B. Run standard baseline attention for the initial prompt tokens. + + // C. Update the global layer index to track progress through the model. + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_is_prefill = false; // Transition to Decode phase for the next token + } + } else { + // --- DECODE PHASE --- + // During decode, we only process the single newest token (T = 1). + + // A. Store the single new Key and Value for this step into the cache at g_current_pos. + + // B. Run specialized Decode Attention. + // This requires reading the history of Keys and Values from the cache + // to compute the output for the current token. + // TODO: Don't forget to scale your attention scores + // TODO: Ensure the output shapes match the next layer + + // C. Update the global state. + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_current_pos++; // Advance the sequence position + } + } +} + +#endif // __ATTENTION_KV_CACHE_TEMPLATE_CUH__ diff --git a/kernels/encoder.cuh b/kernels/encoder.cuh new file mode 100644 index 0000000..351ab5c --- /dev/null +++ b/kernels/encoder.cuh @@ -0,0 +1,31 @@ +#ifndef ENCODER_FORWARD_KERNEL_CUH +#define ENCODER_FORWARD_KERNEL_CUH + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void encoder_forward_kernel(float* out, const int* inp, const float* wte, const float* wpe, + int B, int T, int C) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + int y = blockDim.y * blockIdx.y + threadIdx.y; + + if(x < B && y < T){ + int token = inp[x*T + y]; + for(int i = 0; i < C; i ++){ + out[x*T*C + y*C + i] = wte[token*C+i] + wpe[y*C+i]; + } + } +} + +// Launch kernel here +void encoder_forward(float* out, const int* inp, const float* wte, const float* wpe, int B, int T, int C) { + // Implement this + dim3 block(16,16,1); + dim3 grid((B + block.x -1)/ block.x,(T+block.y-1)/block.y); + + encoder_forward_kernel<<>>(out, inp, wte, wpe,B,T,C); +} + + +#endif // ENCODER_FORWARD_KERNEL_CUH \ No newline at end of file diff --git a/kernels/gelu.cuh b/kernels/gelu.cuh new file mode 100644 index 0000000..311bf4b --- /dev/null +++ b/kernels/gelu.cuh @@ -0,0 +1,32 @@ +#ifndef GELU_KERNEL_CUH_ +#define GELU_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +#define GELU_SCALING_FACTOR sqrtf(2.0f / M_PI) + +__global__ void gelu_forward_kernel(float* out, const float* inp, int N) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + float val = inp[x]; + float cube = 0.044715f * val * val * val; + out[x] = 0.5f * val * (1.0f + tanhf(GELU_SCALING_FACTOR * (val + cube))); + } + +} + +// Launch kernel here +void gelu_forward(float* out, const float* inp, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x -1) / block.x); + + gelu_forward_kernel<<>>(out, inp, N); + +} + +#endif // GELU_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels/layernorm.cuh b/kernels/layernorm.cuh new file mode 100644 index 0000000..0e54094 --- /dev/null +++ b/kernels/layernorm.cuh @@ -0,0 +1,48 @@ +#ifndef __LAYERNORM_KERNEL_CUH__ +#define __LAYERNORM_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void layernorm_forward_kernel(float* out, float* mean, float* rstd, const float* inp, const float* weight, + const float* bias, int B, int T, int C) { + // Implement this + int b = blockDim.x * blockIdx.x + threadIdx.x; + int t = blockDim.y * blockIdx.y + threadIdx.y; + + if (b >= B || t >= T) { + return; + } + + float sum = 0.0f; + for (int c = 0; c < C; ++c) { + sum += inp[b * T * C + t * C + c]; + } + float m = sum / C; + + sum = 0.0f; + for (int c = 0; c < C; ++c) { + float diff = inp[b * T * C + t * C + c] - m; + sum += diff * diff; + } + float s = rsqrtf(sum / C + 1e-5f); + + for (int c = 0; c < C; ++c) { + float n = s * (inp[b * T * C + t * C + c] - m); + out[b * T * C + t * C + c] = n * weight[c] + bias[c]; + } +} + +// Launch kernel here +void layernorm_forward(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + // Implement this + dim3 block(16,16); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y); + + layernorm_forward_kernel<<>>(out,mean, rstd, inp, weight, bias,B,T, C); +} + +#endif // __LAYERNORM_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels/matmul.cuh b/kernels/matmul.cuh new file mode 100644 index 0000000..29634ec --- /dev/null +++ b/kernels/matmul.cuh @@ -0,0 +1,37 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void matmul_forward_kernel(float* out, const float* inp, const float* weight, + const float* bias,int B,int T, int C, int OC) { + // Implement this + int b = blockDim.x*blockIdx.x + threadIdx.x; // B + int t = blockDim.y*blockIdx.y + threadIdx.y; // T + int oc = blockDim.z*blockIdx.z + threadIdx.z; // OC + + if (b >= B || t >= T || oc >= OC) { + return; + } + + float sum = (bias != NULL) ? bias[oc] : 0.0f; + const float* inp_token_ptr = inp + b * T * C + t * C; + for(int i = 0; i < C; i++){ + sum += inp_token_ptr[i] * weight[oc*C + i] ; + } + out[b * T * OC + t * OC + oc] = sum; +} + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + dim3 block(8,8,8); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y, (OC+block.z-1)/block.z); + + matmul_forward_kernel<<>>(out, inp, weight, bias,B,T, C, OC); + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels/residual.cuh b/kernels/residual.cuh new file mode 100644 index 0000000..38a5150 --- /dev/null +++ b/kernels/residual.cuh @@ -0,0 +1,25 @@ +#ifndef RESIDUAL_KERNEL_CUH_ +#define RESIDUAL_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void residual_forward_kernel(float* out, float* inp1, float* inp2, int N) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + out[x] = inp1[x] + inp2[x]; + } +} + +// Launch kernel here +void residual_forward(float* out, float* inp1, float* inp2, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x - 1)/ block.x); + + residual_forward_kernel<<>>(out, inp1, inp2, N); +} + +#endif // RESIDUAL_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels/softmax.cuh b/kernels/softmax.cuh new file mode 100644 index 0000000..7ebb19f --- /dev/null +++ b/kernels/softmax.cuh @@ -0,0 +1,36 @@ +#ifndef __SOFTMAX_KERNEL_CUH__ +#define __SOFTMAX_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void softmax_forward_kernel(float* out, float inv_temperature, const float* inp, int N, int T) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N*T){ + int own_pos = x % T; + const float* a = inp + x * T; + + float maxval = -FLT_MAX; + for (int i = 0; i <= own_pos; i++) { + maxval = fmaxf(maxval,a[i]); + } + + float sumval = 0.0f; + for (int i = 0; i <= own_pos; ++i) { + float ev = expf(inv_temperature * (a[i] - maxval)); + sumval += ev; + out[x * T + i] = ev; + } + + float norm = 1.0f / sumval; + for (int i = 0; i <= own_pos; ++i) { + out[x * T + i] *= norm; + } + } +} + +#endif // __SOFTMAX_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_base/attention.cuh b/kernels_base/attention.cuh new file mode 100644 index 0000000..13df1f9 --- /dev/null +++ b/kernels_base/attention.cuh @@ -0,0 +1,105 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B* T* NH* T){ + int b = (x / T / T / NH); + int nh = (x / T / T) % NH; + int t1 = (x / T) % T; + int t2 = x % T; + + float sum = 0.0f; + for (int i = 0; i < HS; i++) { + sum += q[b*NH*T*HS + nh*T*HS + t1*HS + i] * k[b*NH*T*HS + nh*T*HS + t2*HS + i]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } +} + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + float sum = 0.0f; + for (int t2 = 0; t2 < T; t2++) { + sum += att[b*NH*T*T + nh*T*T + t*T + t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; + } + + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x-1) / block.x,1,1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_base/attention_kv_cache_template.cuh b/kernels_base/attention_kv_cache_template.cuh new file mode 100644 index 0000000..eb9c490 --- /dev/null +++ b/kernels_base/attention_kv_cache_template.cuh @@ -0,0 +1,86 @@ +#ifndef __ATTENTION_KV_CACHE_TEMPLATE_CUH__ +#define __ATTENTION_KV_CACHE_TEMPLATE_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +extern bool g_enable_kv_cache; // Toggles KV-caching logic +extern float* g_kv_cache; // Pointer to the GPU-allocated KV cache buffer +extern int g_layer_idx; // Tracks the current transformer layer, useful for knowing when to switch to decode +extern bool g_is_prefill; // True during the initial prompt, False during token generation +extern int g_current_pos; // Tracks the absolute sequence position during Decode, helps index into the KV cache + +// --- Suggested Kernel Placeholders (TODO: Add your regular attention kernel here, and any other kernels you may need) --- + +/** + * Kernel to store the newly computed Key and Value into the global KV cache. + * CAN BE FUSED WITH QKV MATMUL + * In Prefill: Stores T tokens for the current layer. + * In Decode: Stores 1 token at g_current_pos for the current layer. + */ +__global__ void store_kv_kernel(/* args */) { + // Implement logic to copy K and V from the input QKV buffer into g_kv_cache +} + +/** + * Kernel to compute the attention for the Decode phase (T=1). + * This involves: + * 1. Extracting the current Query (Q) + * 2. Gathering all previous Keys (K) and Values (V) from the cache + * 3. Computing attention using the new Q and cached KV values. + */ +__global__ void decode_attention_kernel(/* args */) { + // Implement T=1 attention logic using the KV cache +} + +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // 1. Check if KV-caching is enabled, used by the verification script + if (!g_enable_kv_cache) { + // TODO: Call your standard Baseline Attention here (Milestone 1/2) + return; + } + + // 2. Initialize KV Cache if necessary + if (g_kv_cache == nullptr) { + // TODO: Allocate the KV cache, based accordingly to the model parameters + // Option 1: Allocate a very big kv cache to store all the tokens + // Option 2: Dynamically resize the kv cache repeatedly for each new token + } + + if (g_is_prefill) { + // --- PREFILL PHASE --- + // During prefill, we process the entire input sequence (T > 1). + + // A. Store the K and V results from this layer into the cache. + // The projections are available in 'inp' (the QKV matmul output). + + // B. Run standard baseline attention for the initial prompt tokens. + + // C. Update the global layer index to track progress through the model. + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_is_prefill = false; // Transition to Decode phase for the next token + } + } else { + // --- DECODE PHASE --- + // During decode, we only process the single newest token (T = 1). + + // A. Store the single new Key and Value for this step into the cache at g_current_pos. + + // B. Run specialized Decode Attention. + // This requires reading the history of Keys and Values from the cache + // to compute the output for the current token. + // TODO: Don't forget to scale your attention scores + // TODO: Ensure the output shapes match the next layer + + // C. Update the global state. + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_current_pos++; // Advance the sequence position + } + } +} + +#endif // __ATTENTION_KV_CACHE_TEMPLATE_CUH__ diff --git a/kernels_base/encoder.cuh b/kernels_base/encoder.cuh new file mode 100644 index 0000000..351ab5c --- /dev/null +++ b/kernels_base/encoder.cuh @@ -0,0 +1,31 @@ +#ifndef ENCODER_FORWARD_KERNEL_CUH +#define ENCODER_FORWARD_KERNEL_CUH + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void encoder_forward_kernel(float* out, const int* inp, const float* wte, const float* wpe, + int B, int T, int C) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + int y = blockDim.y * blockIdx.y + threadIdx.y; + + if(x < B && y < T){ + int token = inp[x*T + y]; + for(int i = 0; i < C; i ++){ + out[x*T*C + y*C + i] = wte[token*C+i] + wpe[y*C+i]; + } + } +} + +// Launch kernel here +void encoder_forward(float* out, const int* inp, const float* wte, const float* wpe, int B, int T, int C) { + // Implement this + dim3 block(16,16,1); + dim3 grid((B + block.x -1)/ block.x,(T+block.y-1)/block.y); + + encoder_forward_kernel<<>>(out, inp, wte, wpe,B,T,C); +} + + +#endif // ENCODER_FORWARD_KERNEL_CUH \ No newline at end of file diff --git a/kernels_base/gelu.cuh b/kernels_base/gelu.cuh new file mode 100644 index 0000000..311bf4b --- /dev/null +++ b/kernels_base/gelu.cuh @@ -0,0 +1,32 @@ +#ifndef GELU_KERNEL_CUH_ +#define GELU_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +#define GELU_SCALING_FACTOR sqrtf(2.0f / M_PI) + +__global__ void gelu_forward_kernel(float* out, const float* inp, int N) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + float val = inp[x]; + float cube = 0.044715f * val * val * val; + out[x] = 0.5f * val * (1.0f + tanhf(GELU_SCALING_FACTOR * (val + cube))); + } + +} + +// Launch kernel here +void gelu_forward(float* out, const float* inp, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x -1) / block.x); + + gelu_forward_kernel<<>>(out, inp, N); + +} + +#endif // GELU_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels_base/layernorm.cuh b/kernels_base/layernorm.cuh new file mode 100644 index 0000000..0e54094 --- /dev/null +++ b/kernels_base/layernorm.cuh @@ -0,0 +1,48 @@ +#ifndef __LAYERNORM_KERNEL_CUH__ +#define __LAYERNORM_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void layernorm_forward_kernel(float* out, float* mean, float* rstd, const float* inp, const float* weight, + const float* bias, int B, int T, int C) { + // Implement this + int b = blockDim.x * blockIdx.x + threadIdx.x; + int t = blockDim.y * blockIdx.y + threadIdx.y; + + if (b >= B || t >= T) { + return; + } + + float sum = 0.0f; + for (int c = 0; c < C; ++c) { + sum += inp[b * T * C + t * C + c]; + } + float m = sum / C; + + sum = 0.0f; + for (int c = 0; c < C; ++c) { + float diff = inp[b * T * C + t * C + c] - m; + sum += diff * diff; + } + float s = rsqrtf(sum / C + 1e-5f); + + for (int c = 0; c < C; ++c) { + float n = s * (inp[b * T * C + t * C + c] - m); + out[b * T * C + t * C + c] = n * weight[c] + bias[c]; + } +} + +// Launch kernel here +void layernorm_forward(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + // Implement this + dim3 block(16,16); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y); + + layernorm_forward_kernel<<>>(out,mean, rstd, inp, weight, bias,B,T, C); +} + +#endif // __LAYERNORM_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_base/matmul.cuh b/kernels_base/matmul.cuh new file mode 100644 index 0000000..29634ec --- /dev/null +++ b/kernels_base/matmul.cuh @@ -0,0 +1,37 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void matmul_forward_kernel(float* out, const float* inp, const float* weight, + const float* bias,int B,int T, int C, int OC) { + // Implement this + int b = blockDim.x*blockIdx.x + threadIdx.x; // B + int t = blockDim.y*blockIdx.y + threadIdx.y; // T + int oc = blockDim.z*blockIdx.z + threadIdx.z; // OC + + if (b >= B || t >= T || oc >= OC) { + return; + } + + float sum = (bias != NULL) ? bias[oc] : 0.0f; + const float* inp_token_ptr = inp + b * T * C + t * C; + for(int i = 0; i < C; i++){ + sum += inp_token_ptr[i] * weight[oc*C + i] ; + } + out[b * T * OC + t * OC + oc] = sum; +} + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + dim3 block(8,8,8); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y, (OC+block.z-1)/block.z); + + matmul_forward_kernel<<>>(out, inp, weight, bias,B,T, C, OC); + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_base/residual.cuh b/kernels_base/residual.cuh new file mode 100644 index 0000000..38a5150 --- /dev/null +++ b/kernels_base/residual.cuh @@ -0,0 +1,25 @@ +#ifndef RESIDUAL_KERNEL_CUH_ +#define RESIDUAL_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void residual_forward_kernel(float* out, float* inp1, float* inp2, int N) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + out[x] = inp1[x] + inp2[x]; + } +} + +// Launch kernel here +void residual_forward(float* out, float* inp1, float* inp2, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x - 1)/ block.x); + + residual_forward_kernel<<>>(out, inp1, inp2, N); +} + +#endif // RESIDUAL_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels_base/softmax.cuh b/kernels_base/softmax.cuh new file mode 100644 index 0000000..7ebb19f --- /dev/null +++ b/kernels_base/softmax.cuh @@ -0,0 +1,36 @@ +#ifndef __SOFTMAX_KERNEL_CUH__ +#define __SOFTMAX_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void softmax_forward_kernel(float* out, float inv_temperature, const float* inp, int N, int T) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N*T){ + int own_pos = x % T; + const float* a = inp + x * T; + + float maxval = -FLT_MAX; + for (int i = 0; i <= own_pos; i++) { + maxval = fmaxf(maxval,a[i]); + } + + float sumval = 0.0f; + for (int i = 0; i <= own_pos; ++i) { + float ev = expf(inv_temperature * (a[i] - maxval)); + sumval += ev; + out[x * T + i] = ev; + } + + float norm = 1.0f / sumval; + for (int i = 0; i <= own_pos; ++i) { + out[x * T + i] *= norm; + } + } +} + +#endif // __SOFTMAX_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_op_11/attention.cuh b/kernels_op_11/attention.cuh new file mode 100644 index 0000000..b40b29c --- /dev/null +++ b/kernels_op_11/attention.cuh @@ -0,0 +1,258 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + + +//added templates so the compiler can unroll loops and optimize better. +// op11 +// without swilling the QK^T inner loops read smem_q [query_row][i] where each thread query_row in teh warp holds a different row but reads teh same column i. as HS is 64 every thread bank index reduces to (i % 32) giving 32 way back conflict per warp on the Q reads. col -> col ^ row to swizzle the accesses and reduce bank conflicts. so each 32 read to 32 different banks instead of 32 reads to the same bank. +template +__global__ void flash_attention_forward_kernel(const float* __restrict__ Q, const float* __restrict__ K, const float* __restrict__ V, float* __restrict__ out, int B, int T, int NH, float scale) { + //following line by like psudo code from flash attention paper. + // https://tridao.me/publications/flash2/flash2.pdf + + // step 1 map thread blocks to (batch , head , query_tile) + int b = blockIdx.x; + int nh = blockIdx.y; + int query_tile = blockIdx.z; + + int query_row = threadIdx.x; //load index + int query_global = query_tile * BR + query_row; + + //step 2 calculate all the base pointers for this block. We will be loading queries from Q and keys/values from K and V, and writing output to out. + const float* __restrict__ query_base = Q + b * NH * T * HS + nh * T * HS; + const float* __restrict__ key_base = K + b * NH * T * HS + nh * T * HS; + const float* __restrict__ value_base = V + b * NH * T * HS + nh * T * HS; + float* out_base = out + b * NH * T * HS + nh * T * HS; + + // declare shared memory for the tile of queries and keys/values. We will load a tile of queries and then iterate over tiles of keys/values. + __shared__ float smem_q[BR * HS]; + __shared__ float smem_k[BC * HS]; + __shared__ float smem_v[BC * HS]; + + // we will compute the output for a tile of queries at a time. + int query_tile_start = query_tile * BR; // global index of the first query in this tile. This is used to load the correct queries into shared memory and also to know which output rows we are responsible for computing. + bool q_valid = (query_global < T); // check if this thread's query is within the sequence length. If not, it will do nothing. + + + //define swizzling function using the XOR on col and row index to reduce bank conflicts on smem_q reads in the inner loop. + + #define SWIZZLE(col, row) ((row) * HS + ((col) ^ ((row) & (HS - 1)))) + + // step 4 load query tiles from global memory to shared memory. + // coalesced load of the queries for this tile into shared memory. a linear load of BR*HS elements, with each thread loading HS elements for one query. + #pragma unroll + for (int i = 0; i < HS; i++) { + smem_q[SWIZZLE(i, query_row)] = (query_tile_start + query_row < T) ? query_base[(query_tile_start + query_row) * HS + i] : 0.0f; // if the query is out of bounds, we can load zeros since they won't contribute to the output. + } + + // step 5 init accumulators for the output of this tile of queries. + float out_reg[HS]; + #pragma unroll + for (int i = 0; i < HS; i++) { + out_reg[i] = 0.0f; + } + float max_score = -FLT_MAX; + float sum_exp = 0.0f; + + + // step 6 determine KV tile range for casual masking + int query_tile_end = (query_tile + 1) * BR; + int n_kv_tiles = (query_tile_end < T) ? (T + BC - 1) / BC : (query_tile_end + BC - 1) / BC; // if the end of the query tile is within the sequence length, we only need to iterate over the KV tiles that cover the sequence length. If the end of the query tile exceeds the sequence length, we need to iterate over all KV tiles that cover the end of the query tile to properly apply casual masking. + + __syncthreads(); + + //step 7 main loop over kv values + for(int j = 0; j < n_kv_tiles; j++){ + __syncthreads(); + + int k_tile_start = j * BC; + + // step 8 load k and v tiles from global memory + #pragma unroll + for (int i = 0; i < HS; i++) { + int l = i * BC + query_row; + int row = l / HS; + int col = l % HS; + int global_row = k_tile_start + row; + bool valid = (global_row < T) && (row < BC); + + if (valid) { + smem_k[SWIZZLE(col, row)] = key_base[global_row * HS + col]; + smem_v[SWIZZLE(col, row)] = value_base[global_row * HS + col]; + } else { + smem_k[SWIZZLE(col, row)] = 0.0f; // pad with zeros if out of bounds + smem_v[SWIZZLE(col, row)] = 0.0f; // pad with zeros if out of bounds + } + } + __syncthreads(); + + // step 9 compute one row of S = Qi * Kj^T and apply casual masking + float score_reg[BC]; + float row_max = -FLT_MAX; + + #pragma unroll + for (int c = 0; c < BC; c++) { + int k_global = j * BC + c; + float dot; // compute the dot product of the query and key for this position + + if (!q_valid || k_global > query_global) { // apply casual masking. If the key position is greater than the query position, we mask it out by setting the score to a very large negative number. + dot = -FLT_MAX; + } else { + float acc = 0.0f; + #pragma unroll + for (int h = 0; h < HS; h++) { + acc += smem_q[SWIZZLE(h, query_row)] * smem_k[SWIZZLE(h, c)]; + } + dot = acc * scale; + } + score_reg[c] = dot; + row_max = fmaxf(row_max, dot); // local max for this row + } + + // step 10 online softmax rescaling + float max_new = fmaxf(max_score, row_max); + if (max_new == -FLT_MAX) { // if both are -FLT_MAX, we can skip the rescaling because all scores are -FLT_MAX and the output will be zero anyway. + continue; + } + float alpha = (max_score == -FLT_MAX) ? 0.0f : expf(max_score - max_new); // if max_score is -FLT_MAX, we set alpha to 0 to avoid exp overflow. Otherwise, we compute the scaling factor for the old scores. + float local_tile = 0.0f; // this will hold the sum of exp for the scores in this tile after rescaling. + + #pragma unroll + for (int c = 0; c < BC; c++) { + float p = (score_reg[c] == -FLT_MAX) ? 0.0f : expf(score_reg[c] - max_new); // compute the exp for this score after rescaling. If the score is -FLT_MAX, we set it to 0 to avoid exp underflow. + local_tile += p; + score_reg[c] = p; // store the rescaled exp in score_reg for later use in the weighted sum. + } + + // update output acc + #pragma unroll + for (int h = 0; h < HS; ++h) { + float pv = 0.0f; + #pragma unroll + for (int c = 0; c < BC; ++c) { + pv += score_reg[c] * smem_v[SWIZZLE(h, c)]; + } + out_reg[h] = alpha * out_reg[h] + pv; + } + sum_exp = alpha * sum_exp + local_tile; // update the local tile sum with the contribution from this tile and the previous tiles, scaled by alpha. + max_score = max_new; // update the max score for the next iteration + + } + if (q_valid) { + float inv_l = (sum_exp > 0.0f) ? (1.0f / sum_exp) : 0.0f; + #pragma unroll + for (int h = 0; h < HS; ++h) { + out_base[query_global * HS + h] = out_reg[h] * inv_l; + } + } +} + + + +__global__ void permute_kernel(float* __restrict__ q, float* __restrict__ k, float* __restrict__ v, const float* __restrict__ inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(const float* __restrict__ inp, float* __restrict__ out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +// __global__ void preattention_scores_kernel(const float* __restrict__ q, const float* __restrict__ k, int B, int T, int NH, int HS, float* __restrict__ preatt){ +// int x = blockDim.x * blockIdx.x + threadIdx.x; + +// if(x < B* T* NH* T){ +// int b = (x / T / T / NH); +// int nh = (x / T / T) % NH; +// int t1 = (x / T) % T; +// int t2 = x % T; + +// float sum = 0.0f; +// const float* __restrict__ q_ptr = q + b * NH * T * HS + nh * T * HS + t1 * HS; +// const float* __restrict__ k_ptr = k + b * NH * T * HS + nh * T * HS + t2 * HS; +// for (int i = 0; i < HS; i++) { +// sum += q_ptr[i] * k_ptr[i]; +// } +// preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; +// } +// } + +// __global__ void attention_v_kernel(float* __restrict__ vaccum, const float* __restrict__ att, const float* __restrict__ v, int B, int T, int NH, int HS){ +// int x = blockDim.x * blockIdx.x + threadIdx.x; + +// if(x < B*NH*T*HS){ +// int b = (x / HS / T / NH); +// int nh = (x / HS / T) % NH; +// int t = (x / HS) % T; +// int hs = x % HS; +// float sum = 0.0f; +// const float* __restrict__ att_ptr = att + b * NH * T * T + nh * T * T + t * T; +// for (int t2 = 0; t2 < T; t2++) { +// sum += att_ptr[t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; +// } + +// vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; +// } +// } + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(256,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + //fa2 + constexpr int BR = 64; // tile size for queries + constexpr int BC = 64; // tile size for keys/values + float scale = 1.0f / sqrtf((float)HS); + + int number_of_q_tiles = (T + BR - 1) / BR; + dim3 fa_grid(B, NH, number_of_q_tiles); // one block per (batch, head, Q tile) + dim3 fa_block(BR, 1, 1); // one warp per query in the tile + + float* vaccum = inp; + flash_attention_forward_kernel<<>>(q, k, v, vaccum, B, T, NH, scale); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_op_12/matmul.cuh b/kernels_op_12/matmul.cuh new file mode 100644 index 0000000..9869056 --- /dev/null +++ b/kernels_op_12/matmul.cuh @@ -0,0 +1,57 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +#define TILE_SIZE 32 +#define PADDING 1 + +//op12 +// Adding +1 padding to the inner dimension shifts each row by one bank, distributing the 32 column reads across all 32 banks. + +__global__ void matmul_kernel(const float* __restrict__ inp, float* __restrict__ out, const float* __restrict__ weights, + const float* __restrict__ biases, int B, int T, int OC, int C ) { + // Implement this + // reshape (BT X C) * (OC X C)^T -> (BT X OC) + + int bt = blockIdx.y * TILE_SIZE + threadIdx.y; + int oc = blockIdx.x * TILE_SIZE + threadIdx.x; + int BT = B*T; + + __shared__ float A_smem_tile[TILE_SIZE][TILE_SIZE + PADDING]; + __shared__ float B_smem_tile[TILE_SIZE][TILE_SIZE + PADDING]; + + float acc = 0.0f; + int num_tiles = (C + TILE_SIZE - 1) / TILE_SIZE; + + for(int t = 0; t < num_tiles; t++){ + int c = t * TILE_SIZE + threadIdx.x; + int c_ = t * TILE_SIZE + threadIdx.y; + A_smem_tile[threadIdx.y][threadIdx.x] = (bt < BT && c < C) ? inp[bt * C + c] : 0.0f; + B_smem_tile[threadIdx.y][threadIdx.x] = (oc < OC && c_ < C) ? weights[oc * C + c_] : 0.0f; + + __syncthreads(); + + #pragma unroll + for(int i = 0; i < TILE_SIZE; i++){ + acc += A_smem_tile[threadIdx.y][i] * B_smem_tile[i][threadIdx.x]; + } + __syncthreads(); + } + if(bt < BT && oc < OC){ + if(biases != nullptr){ + acc += biases[oc]; + } + out[bt * OC + oc] = acc; + } +} + +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC){ + dim3 block(TILE_SIZE, TILE_SIZE); + dim3 grid((OC + TILE_SIZE - 1) / TILE_SIZE, (B * T + TILE_SIZE - 1) / TILE_SIZE); + matmul_kernel<<>>(inp, out, weight, bias, B, T, OC, C); +} + +#endif \ No newline at end of file diff --git a/kernels_op_13/gpt2.cuh b/kernels_op_13/gpt2.cuh new file mode 100644 index 0000000..0604d18 --- /dev/null +++ b/kernels_op_13/gpt2.cuh @@ -0,0 +1,417 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include +#include +#include +#include +#include +#include +#include +#include + +// GPU / CUDA related +#include +#include +#include "utils/utils.h" +#include "utils/tokenizer.h" +#include "utils/cuda_utils.cuh" + +// kernels +#include "kernels/attention.cuh" +#include "kernels/encoder.cuh" +#include "kernels/gelu.cuh" +#include "kernels/layernorm.cuh" +#include "kernels/matmul.cuh" +#include "kernels/residual.cuh" + +// ---------------------------------------------------------------------------- +// GPT-2 model definition + +typedef struct { + int max_seq_len; + int vocab_size; + int padded_vocab_size; // padded to multiple of 128 + int num_layers; + int num_heads; + int channels; +} GPT2Config; + +// the parameters of the model +#define NUM_PARAMETER_TENSORS 16 +typedef struct { + float* wte; // (Vp, C) + float* wpe; // (maxT, C) + float* ln1w; // (L, C) + float* ln1b; // (L, C) + float* qkvw; // (L, 3*C, C) + float* qkvb; // (L, 3*C) + float* attprojw; // (L, C, C) + float* attprojb; // (L, C) + float* ln2w; // (L, C) + float* ln2b; // (L, C) + float* fcw; // (L, 4*C, C) + float* fcb; // (L, 4*C) + float* fcprojw; // (L, C, 4*C) + float* fcprojb; // (L, C) + float* lnfw; // (C) + float* lnfb; // (C) +} ParameterTensors; + +void fill_in_parameter_sizes(size_t* param_sizes, GPT2Config config) { + int Vp = config.padded_vocab_size; + int C = config.channels; + int maxT = config.max_seq_len; + int L = config.num_layers; + param_sizes[0] = Vp * C; // wte + param_sizes[1] = maxT * C; // wpe + param_sizes[2] = L * C; // ln1w + param_sizes[3] = L * C; // ln1b + param_sizes[4] = L * (3 * C) * C; // qkvw + param_sizes[5] = L * (3 * C); // qkvb + param_sizes[6] = L * C * C; // attprojw + param_sizes[7] = L * C; // attprojb + param_sizes[8] = L * C; // ln2w + param_sizes[9] = L * C; // ln2b + param_sizes[10] = L * (4 * C) * C; // fcw + param_sizes[11] = L * (4 * C); // fcb + param_sizes[12] = L * C * (4 * C); // fcprojw + param_sizes[13] = L * C; // fcprojb + param_sizes[14] = C; // lnfw + param_sizes[15] = C; // lnfb +} + +// allocate memory for the parameters and point the individual tensors to the right places +float* malloc_and_point_parameters(ParameterTensors* params, size_t* param_sizes, int on_device) { + // on_device: 0 = CPU, 1 = GPU + // calculate the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += param_sizes[i]; + } + // malloc all parameters all at once on the device + float* params_memory; + if (on_device) { + cudaCheck(cudaMalloc((void**)¶ms_memory, num_parameters * sizeof(float))); + } else { + params_memory = (float*)mallocCheck(num_parameters * sizeof(float)); + } + // assign all the tensors their place in the array + float** ptrs[] = { + ¶ms->wte, ¶ms->wpe, ¶ms->ln1w, ¶ms->ln1b, ¶ms->qkvw, ¶ms->qkvb, + ¶ms->attprojw, ¶ms->attprojb, ¶ms->ln2w, ¶ms->ln2b, ¶ms->fcw, ¶ms->fcb, + ¶ms->fcprojw, ¶ms->fcprojb, ¶ms->lnfw, ¶ms->lnfb + }; + float* params_memory_iterator = params_memory; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + *(ptrs[i]) = params_memory_iterator; + params_memory_iterator += param_sizes[i]; + } + return params_memory; +} + +#define NUM_ACTIVATION_TENSORS 21 +typedef struct { + float* encoded; // (B, T, C) + float* ln1; // (L, B, T, C) + float* ln1_mean; // (L, B, T) + float* ln1_rstd; // (L, B, T) + float* atty; // (L, B, T, C) + float* att; // (L, B, NH, T, T) + float* attproj; // (L, B, T, C) + float* residual2; // (L, B, T, C) + float* ln2; // (L, B, T, C) + float* ln2_mean; // (L, B, T) + float* ln2_rstd; // (L, B, T) + float* fch; // (L, B, T, 4*C) + float* fch_gelu; // (L, B, T, 4*C) + float* fcproj; // (L, B, T, C) + float* residual3; // (L, B, T, C) + float* lnf; // (B, T, C) + float* lnf_mean; // (B, T) + float* lnf_rstd; // (B, T) + float* losses; // (B, T) + float* qkvr; // (L, B, T, 3*C) + // This buffer will store the logits + // during the processing of transformer blocks, we will also use this as a + // general scratchpad buffer. Allocation is made large enough to hold (B, T, 3C), + // (B, NH, T, T), and (B, T, V) shaped tensors. + float* output; +} ActivationTensors; + +void fill_in_activation_sizes(size_t* act_sizes, int B, int T, GPT2Config config) { + size_t Vp = config.padded_vocab_size; + size_t L = config.num_layers; + size_t NH = config.num_heads; + size_t C = config.channels; + act_sizes[0] = B * T * C; // encoded + act_sizes[1] = L * B * T * C; // ln1 + act_sizes[2] = 1; // ln1_mean, reserved for future use + act_sizes[3] = 1; // ln1_rstd, reserved for future use + act_sizes[4] = L * B * T * C; // atty + act_sizes[5] = L * B * NH * T * T; // att + act_sizes[6] = L * B * T * C; // attproj + act_sizes[7] = L * B * T * C; // residual2 + act_sizes[8] = L * B * T * C; // ln2 + act_sizes[9] = 1; // ln2_mean, reserved for future use + act_sizes[10] = 1; // ln2_rstd, reserved for future use + act_sizes[11] = L * B * T * 4*C; // fch + act_sizes[12] = L * B * T * 4*C; // fch_gelu + act_sizes[13] = L * B * T * C; // fcproj + act_sizes[14] = L * B * T * C; // residual3 + act_sizes[15] = B * T * C; // lnf + act_sizes[16] = 1; // lnf_mean, reserved for future use + act_sizes[17] = 1; // lnf_rstd, reserved for future use + act_sizes[18] = B * T; // losses + act_sizes[19] = L * B * T * 3*C; // qkvr + act_sizes[20] = B * T * max(3*C, max(NH*T, Vp)); // output / scratch +} + +// Reserved for future use +#define NUM_BACKWARD_TENSORS 3 +typedef struct { + float* bt4c; // (B, T, 4*C) + float* preatt; // (B, NH, T, T) + float* residual3; // (B, T, C) +} GradActTensors; + +//op13 +//aligning to 256 byte, so each tensor gtes its own perfectly aligned memory region. +#define align_to_256_bytes(x) (x + (256/sizeof(float) - (x % (256/sizeof(float)))) % (256/sizeof(float))) + +float* malloc_and_point(float** targets[], const size_t* act_sizes, int n) { + size_t num_activations = 0; + for (size_t i = 0; i < n; i++) { + num_activations += align_to_256_bytes(act_sizes[i]); + } + float* acts_memory; + cudaCheck(cudaMalloc((void**)&acts_memory, num_activations * sizeof(float))); + // clear the allocated memory since cudaMalloc does not guarantee zeroed memory + cudaCheck(cudaMemset(acts_memory, 0, num_activations * sizeof(float))); + float* acts_memory_iterator = acts_memory; + for (size_t i = 0; i < n; i++) { + *(targets[i]) = acts_memory_iterator; + acts_memory_iterator += align_to_256_bytes(act_sizes[i]); + } + return acts_memory; +} + +float* malloc_and_point_activations(ActivationTensors* acts, const size_t* act_sizes) { + float** ptrs[] = { + &acts->encoded, &acts->ln1, &acts->ln1_mean, &acts->ln1_rstd, &acts->atty, + &acts->att, &acts->attproj, &acts->residual2, &acts->ln2, &acts->ln2_mean, + &acts->ln2_rstd, &acts->fch, &acts->fch_gelu, &acts->fcproj, &acts->residual3, &acts->lnf, + &acts->lnf_mean, &acts->lnf_rstd, &acts->losses, &acts->qkvr, &acts->output + }; + return malloc_and_point(ptrs, act_sizes, NUM_ACTIVATION_TENSORS); +} + +typedef struct { + GPT2Config config; + // the weights of the model, and their sizes + ParameterTensors params; + size_t param_sizes[NUM_PARAMETER_TENSORS]; + float* params_memory; + size_t num_parameters; + ParameterTensors grads; + float* grads_memory; + float* m_memory; + float* v_memory; + // the activations of the model, and their sizes + ActivationTensors acts; + size_t act_sizes[NUM_ACTIVATION_TENSORS]; + float* acts_memory; + size_t num_activations; + GradActTensors grads_acts; + size_t num_grad_acts; + float* grads_acts_memory; + // other run state configuration + int batch_size; // the batch size (B) of current forward pass + int seq_len; // the sequence length (T) of current forward pass + int* inputs; // the input tokens for the current forward pass + int* targets; // the target tokens for the current forward pass + float mean_loss; + float* cpu_losses; +} GPT2; + +void gpt2_build_from_checkpoint(GPT2 *model, const char* checkpoint_path) { + + // read in model from a checkpoint file + FILE *model_file = fopenCheck(checkpoint_path, "rb"); + int model_header[256]; + freadCheck(model_header, sizeof(int), 256, model_file); + if (model_header[0] != 20240326) { fprintf(stderr, "GPT2 - Bad magic model file\n"); exit(EXIT_FAILURE); } + if (model_header[1] != 3) { + // was bumped from 1 -> 3 to incorporate the padded vocab size + fprintf(stderr, "GPT2 - Bad version in model file\n"); + exit(EXIT_FAILURE); + } + + // read in hyperparameters + model->config.max_seq_len = model_header[2]; + model->config.vocab_size = model_header[3]; + model->config.num_layers = model_header[4]; + model->config.num_heads = model_header[5]; + model->config.channels = model_header[6]; + model->config.padded_vocab_size = model_header[7]; + + // allocate space for all the parameters and read them in + fill_in_parameter_sizes(model->param_sizes, model->config); + + // count the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += model->param_sizes[i]; + } + model->num_parameters = num_parameters; + + // create memory for model parameters on the device + model->params_memory = malloc_and_point_parameters(&model->params, model->param_sizes, 1); + + // read in all the parameters from file and copy them to device + float* params_memory_cpu = (float*)mallocCheck(num_parameters * sizeof(float)); + freadCheck(params_memory_cpu, sizeof(float), num_parameters, model_file); + cudaCheck(cudaMemcpy(model->params_memory, params_memory_cpu, num_parameters * sizeof(float), cudaMemcpyHostToDevice)); + free(params_memory_cpu); + fcloseCheck(model_file); + + // other inits + model->acts_memory = NULL; + model->grads_memory = NULL; + model->m_memory = NULL; + model->v_memory = NULL; + model->grads_acts_memory = NULL; + model->inputs = NULL; + model->targets = NULL; + model->cpu_losses = NULL; + model->batch_size = 0; + model->seq_len = 0; + model->mean_loss = -1.0f; // -1.0f will designate no loss +} + +void gpt2_forward(GPT2 *model, int* inputs, int B, int T) { + + // ensure the model was initialized or error out + if (model->params_memory == NULL) { + printf("GPT2 - Error: model was not initialized properly.\n"); + exit(EXIT_FAILURE); + } + + // convenience parameters + int V = model->config.vocab_size; + int Vp = model->config.padded_vocab_size; + int L = model->config.num_layers; + int NH = model->config.num_heads; + int C = model->config.channels; + + // validate inputs, all indices must be in the range [0, V) + for(int i = 0; i < B * T; i++) { + assert(0 <= inputs[i] && inputs[i] < V); + } + + // allocate space for all the activations if needed (done here, lazily) + if(model->acts_memory == NULL || B != model->batch_size || T != model->seq_len) { + // record the current B,T as well + model->batch_size = B; + model->seq_len = T; + // and now allocate the space + fill_in_activation_sizes(model->act_sizes, B, T, model->config); + size_t num_activations = 0; + for (size_t i = 0; i < NUM_ACTIVATION_TENSORS; i++) { + num_activations += model->act_sizes[i]; + } + model->num_activations = num_activations; + if (model->acts_memory != NULL) { + printf("GPT2 - Reallocating activations for B=%d T=%d\n", B, T); + cudaFree(model->acts_memory); + cudaFree(model->inputs); + } + model->acts_memory = malloc_and_point_activations(&model->acts, model->act_sizes); + printf("GPT2 - Allocated %zu MiB for activations\n", (num_activations * sizeof(float)) >> 20); // >> 20 is /(1024*1024) + cudaCheck(cudaMalloc((void**)&model->inputs, B * T * sizeof(int))); + cudaCheck(cudaMallocHost((void**)&model->cpu_losses, B * T * sizeof(float))); + } + + // copy inputs to the model + cudaCheck(cudaMemcpy(model->inputs, inputs, B * T * sizeof(int), cudaMemcpyHostToDevice)); + + // forward pass + ParameterTensors params = model->params; // for brevity + ActivationTensors acts = model->acts; + float* residual; + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe, B, T, C); // encoding goes into residual[0] + + for (int l = 0; l < L; l++) { + + residual = l == 0 ? acts.encoded : acts.residual3 + (l-1) * B * T * C; + + // get the pointers of the weights for this layer + float* l_ln1w = params.ln1w + l * C; + float* l_ln1b = params.ln1b + l * C; + float* l_qkvw = params.qkvw + l * 3*C * C; + float* l_qkvb = params.qkvb + l * 3*C; + float* l_attprojw = params.attprojw + l * C * C; + float* l_attprojb = params.attprojb + l * C; + float* l_ln2w = params.ln2w + l * C; + float* l_ln2b = params.ln2b + l * C; + float* l_fcw = params.fcw + l * 4*C * C; + float* l_fcb = params.fcb + l * 4*C; + float* l_fcprojw = params.fcprojw + l * C * 4*C; + float* l_fcprojb = params.fcprojb + l * C; + + // get the pointers of the activations for this layer + float* l_ln1 = acts.ln1 + l * B * T * C; + float* l_ln1_mean = acts.ln1_mean; + float* l_ln1_rstd = acts.ln1_rstd; + float* l_qkvr = acts.qkvr + l * B * T * 3*C; + float* l_atty = acts.atty + l * B * T * C; + float* l_att = acts.att + l * B * NH * T * T; + float* l_attproj = acts.attproj + l * B * T * C; + float* l_residual2 = acts.residual2 + l * B * T * C; + float* l_ln2 = acts.ln2 + l * B * T * C; + float* l_ln2_mean = acts.ln2_mean; + float* l_ln2_rstd = acts.ln2_rstd; + float* l_fch = acts.fch + l * B * T * 4*C; + float* l_fch_gelu = acts.fch_gelu + l * B * T * 4*C; + float* l_fcproj = acts.fcproj + l * B * T * C; + float* l_residual3 = acts.residual3 + l * B * T * C; + // these are only needed as scratchpads for the forward pass + float* scratch = acts.output; + + // now do the forward pass + layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C); + matmul_forward(scratch, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C); + attention_forward(l_atty, l_qkvr, l_att, scratch, B, T, C, NH); + matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C); + residual_forward(l_residual2, residual, l_attproj, B*T*C); + layernorm_forward(l_ln2, l_ln2_mean, l_ln2_rstd, l_residual2, l_ln2w, l_ln2b, B, T, C); + matmul_forward(l_fch, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + gelu_forward(l_fch_gelu, l_fch, B*T*4*C); + matmul_forward(l_fcproj, l_fch_gelu, l_fcprojw, l_fcprojb, B, T, 4*C, C); + residual_forward(l_residual3, l_residual2, l_fcproj, B*T*C); + } + + residual = acts.residual3 + (L-1) * B * T * C; // last residual is in residual3 + layernorm_forward(acts.lnf, acts.lnf_mean, acts.lnf_rstd, residual, params.lnfw, params.lnfb, B, T, C); + matmul_forward(acts.output, acts.lnf, params.wte, NULL, B, T, C, Vp); + + model->mean_loss = -1.0f; +} + +void gpt2_free(GPT2 *model) { + cudaCheck(cudaFree(model->params_memory)); + cudaCheck(cudaFree(model->grads_memory)); + cudaCheck(cudaFree(model->m_memory)); + cudaCheck(cudaFree(model->v_memory)); + cudaCheck(cudaFree(model->acts_memory)); + cudaCheck(cudaFree(model->grads_acts_memory)); + cudaCheck(cudaFree(model->inputs)); + cudaCheck(cudaFree(model->targets)); + cudaFreeHost(model->cpu_losses); +} + +#define GPT2_EOT 50256 \ No newline at end of file diff --git a/kernels_op_16/gpt2.cuh b/kernels_op_16/gpt2.cuh new file mode 100644 index 0000000..79ad65b --- /dev/null +++ b/kernels_op_16/gpt2.cuh @@ -0,0 +1,416 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include +#include +#include +#include +#include +#include +#include +#include + +// GPU / CUDA related +#include +#include +#include "utils/utils.h" +#include "utils/tokenizer.h" +#include "utils/cuda_utils.cuh" + +// kernels +#include "kernels/attention.cuh" +#include "kernels/encoder.cuh" +#include "kernels/gelu.cuh" +#include "kernels/layernorm.cuh" +#include "kernels/matmul.cuh" +#include "kernels/residual.cuh" + +// ---------------------------------------------------------------------------- +// GPT-2 model definition + +typedef struct { + int max_seq_len; + int vocab_size; + int padded_vocab_size; // padded to multiple of 128 + int num_layers; + int num_heads; + int channels; +} GPT2Config; + +// the parameters of the model +#define NUM_PARAMETER_TENSORS 16 +typedef struct { + float* wte; // (Vp, C) + float* wpe; // (maxT, C) + float* ln1w; // (L, C) + float* ln1b; // (L, C) + float* qkvw; // (L, 3*C, C) + float* qkvb; // (L, 3*C) + float* attprojw; // (L, C, C) + float* attprojb; // (L, C) + float* ln2w; // (L, C) + float* ln2b; // (L, C) + float* fcw; // (L, 4*C, C) + float* fcb; // (L, 4*C) + float* fcprojw; // (L, C, 4*C) + float* fcprojb; // (L, C) + float* lnfw; // (C) + float* lnfb; // (C) +} ParameterTensors; + +void fill_in_parameter_sizes(size_t* param_sizes, GPT2Config config) { + int Vp = config.padded_vocab_size; + int C = config.channels; + int maxT = config.max_seq_len; + int L = config.num_layers; + param_sizes[0] = Vp * C; // wte + param_sizes[1] = maxT * C; // wpe + param_sizes[2] = L * C; // ln1w + param_sizes[3] = L * C; // ln1b + param_sizes[4] = L * (3 * C) * C; // qkvw + param_sizes[5] = L * (3 * C); // qkvb + param_sizes[6] = L * C * C; // attprojw + param_sizes[7] = L * C; // attprojb + param_sizes[8] = L * C; // ln2w + param_sizes[9] = L * C; // ln2b + param_sizes[10] = L * (4 * C) * C; // fcw + param_sizes[11] = L * (4 * C); // fcb + param_sizes[12] = L * C * (4 * C); // fcprojw + param_sizes[13] = L * C; // fcprojb + param_sizes[14] = C; // lnfw + param_sizes[15] = C; // lnfb +} + +// allocate memory for the parameters and point the individual tensors to the right places +float* malloc_and_point_parameters(ParameterTensors* params, size_t* param_sizes, int on_device) { + // on_device: 0 = CPU, 1 = GPU + // calculate the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += param_sizes[i]; + } + // malloc all parameters all at once on the device + float* params_memory; + if (on_device) { + cudaCheck(cudaMalloc((void**)¶ms_memory, num_parameters * sizeof(float))); + } else { + params_memory = (float*)mallocCheck(num_parameters * sizeof(float)); + } + // assign all the tensors their place in the array + float** ptrs[] = { + ¶ms->wte, ¶ms->wpe, ¶ms->ln1w, ¶ms->ln1b, ¶ms->qkvw, ¶ms->qkvb, + ¶ms->attprojw, ¶ms->attprojb, ¶ms->ln2w, ¶ms->ln2b, ¶ms->fcw, ¶ms->fcb, + ¶ms->fcprojw, ¶ms->fcprojb, ¶ms->lnfw, ¶ms->lnfb + }; + float* params_memory_iterator = params_memory; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + *(ptrs[i]) = params_memory_iterator; + params_memory_iterator += param_sizes[i]; + } + return params_memory; +} + +#define NUM_ACTIVATION_TENSORS 21 +typedef struct { + float* encoded; // (B, T, C) + float* ln1; // (L, B, T, C) + float* ln1_mean; // (L, B, T) + float* ln1_rstd; // (L, B, T) + float* atty; // (L, B, T, C) + float* att; // (L, B, NH, T, T) + float* attproj; // (L, B, T, C) + float* residual2; // (L, B, T, C) + float* ln2; // (L, B, T, C) + float* ln2_mean; // (L, B, T) + float* ln2_rstd; // (L, B, T) + float* fch; // (L, B, T, 4*C) + float* fch_gelu; // (L, B, T, 4*C) + float* fcproj; // (L, B, T, C) + float* residual3; // (L, B, T, C) + float* lnf; // (B, T, C) + float* lnf_mean; // (B, T) + float* lnf_rstd; // (B, T) + float* losses; // (B, T) + float* qkvr; // (L, B, T, 3*C) + // This buffer will store the logits + // during the processing of transformer blocks, we will also use this as a + // general scratchpad buffer. Allocation is made large enough to hold (B, T, 3C), + // (B, NH, T, T), and (B, T, V) shaped tensors. + float* output; +} ActivationTensors; + +void fill_in_activation_sizes(size_t* act_sizes, int B, int T, GPT2Config config) { + size_t Vp = config.padded_vocab_size; + size_t L = config.num_layers; + size_t NH = config.num_heads; + size_t C = config.channels; + act_sizes[0] = B * T * C; // encoded + act_sizes[1] = L * B * T * C; // ln1 + //to align making 1 to 4 + act_sizes[2] = 4; // ln1_mean, reserved for future use + act_sizes[3] = 4; // ln1_rstd, reserved for future use + act_sizes[4] = L * B * T * C; // atty + act_sizes[5] = L * B * NH * T * T; // att + act_sizes[6] = L * B * T * C; // attproj + act_sizes[7] = L * B * T * C; // residual2 + act_sizes[8] = L * B * T * C; // ln2 + act_sizes[9] = 4; // ln2_mean, reserved for future use + act_sizes[10] = 4; // ln2_rstd, reserved for future use + act_sizes[11] = L * B * T * 4*C; // fch + act_sizes[12] = L * B * T * 4*C; // fch_gelu + act_sizes[13] = L * B * T * C; // fcproj + act_sizes[14] = L * B * T * C; // residual3 + act_sizes[15] = B * T * C; // lnf + act_sizes[16] = 4; // lnf_mean, reserved for future use + act_sizes[17] = 4; // lnf_rstd, reserved for future use + act_sizes[18] = B * T; // losses + act_sizes[19] = L * B * T * 3*C; // qkvr + act_sizes[20] = B * T * max(3*C, max(NH*T, Vp)); // output / scratch +} + +// Reserved for future use +#define NUM_BACKWARD_TENSORS 3 +typedef struct { + float* bt4c; // (B, T, 4*C) + float* preatt; // (B, NH, T, T) + float* residual3; // (B, T, C) +} GradActTensors; + +float* malloc_and_point(float** targets[], const size_t* act_sizes, int n) { + size_t num_activations = 0; + for (size_t i = 0; i < n; i++) { + num_activations += act_sizes[i]; + } + float* acts_memory; + cudaCheck(cudaMalloc((void**)&acts_memory, num_activations * sizeof(float))); + // clear the allocated memory since cudaMalloc does not guarantee zeroed memory + cudaCheck(cudaMemset(acts_memory, 0, num_activations * sizeof(float))); + float* acts_memory_iterator = acts_memory; + for (size_t i = 0; i < n; i++) { + *(targets[i]) = acts_memory_iterator; + acts_memory_iterator += act_sizes[i]; + } + return acts_memory; +} + +float* malloc_and_point_activations(ActivationTensors* acts, const size_t* act_sizes) { + float** ptrs[] = { + &acts->encoded, &acts->ln1, &acts->ln1_mean, &acts->ln1_rstd, &acts->atty, + &acts->att, &acts->attproj, &acts->residual2, &acts->ln2, &acts->ln2_mean, + &acts->ln2_rstd, &acts->fch, &acts->fch_gelu, &acts->fcproj, &acts->residual3, &acts->lnf, + &acts->lnf_mean, &acts->lnf_rstd, &acts->losses, &acts->qkvr, &acts->output + }; + return malloc_and_point(ptrs, act_sizes, NUM_ACTIVATION_TENSORS); +} + +typedef struct { + GPT2Config config; + // the weights of the model, and their sizes + ParameterTensors params; + size_t param_sizes[NUM_PARAMETER_TENSORS]; + float* params_memory; + size_t num_parameters; + ParameterTensors grads; + float* grads_memory; + float* m_memory; + float* v_memory; + // the activations of the model, and their sizes + ActivationTensors acts; + size_t act_sizes[NUM_ACTIVATION_TENSORS]; + float* acts_memory; + size_t num_activations; + GradActTensors grads_acts; + size_t num_grad_acts; + float* grads_acts_memory; + // other run state configuration + int batch_size; // the batch size (B) of current forward pass + int seq_len; // the sequence length (T) of current forward pass + int* inputs; // the input tokens for the current forward pass + int* targets; // the target tokens for the current forward pass + float mean_loss; + float* cpu_losses; +} GPT2; + +void gpt2_build_from_checkpoint(GPT2 *model, const char* checkpoint_path) { + + // read in model from a checkpoint file + FILE *model_file = fopenCheck(checkpoint_path, "rb"); + int model_header[256]; + freadCheck(model_header, sizeof(int), 256, model_file); + if (model_header[0] != 20240326) { fprintf(stderr, "GPT2 - Bad magic model file\n"); exit(EXIT_FAILURE); } + if (model_header[1] != 3) { + // was bumped from 1 -> 3 to incorporate the padded vocab size + fprintf(stderr, "GPT2 - Bad version in model file\n"); + exit(EXIT_FAILURE); + } + + // read in hyperparameters + model->config.max_seq_len = model_header[2]; + model->config.vocab_size = model_header[3]; + model->config.num_layers = model_header[4]; + model->config.num_heads = model_header[5]; + model->config.channels = model_header[6]; + model->config.padded_vocab_size = model_header[7]; + + // allocate space for all the parameters and read them in + fill_in_parameter_sizes(model->param_sizes, model->config); + + // count the number of parameters + size_t num_parameters = 0; + for (size_t i = 0; i < NUM_PARAMETER_TENSORS; i++) { + num_parameters += model->param_sizes[i]; + } + model->num_parameters = num_parameters; + + // create memory for model parameters on the device + model->params_memory = malloc_and_point_parameters(&model->params, model->param_sizes, 1); + + // read in all the parameters from file and copy them to device + float* params_memory_cpu = (float*)mallocCheck(num_parameters * sizeof(float)); + freadCheck(params_memory_cpu, sizeof(float), num_parameters, model_file); + cudaCheck(cudaMemcpy(model->params_memory, params_memory_cpu, num_parameters * sizeof(float), cudaMemcpyHostToDevice)); + free(params_memory_cpu); + fcloseCheck(model_file); + + // other inits + model->acts_memory = NULL; + model->grads_memory = NULL; + model->m_memory = NULL; + model->v_memory = NULL; + model->grads_acts_memory = NULL; + model->inputs = NULL; + model->targets = NULL; + model->cpu_losses = NULL; + model->batch_size = 0; + model->seq_len = 0; + model->mean_loss = -1.0f; // -1.0f will designate no loss +} + +void gpt2_forward(GPT2 *model, int* inputs, int B, int T) { + + // ensure the model was initialized or error out + if (model->params_memory == NULL) { + printf("GPT2 - Error: model was not initialized properly.\n"); + exit(EXIT_FAILURE); + } + + // convenience parameters + int V = model->config.vocab_size; + int Vp = model->config.padded_vocab_size; + int L = model->config.num_layers; + int NH = model->config.num_heads; + int C = model->config.channels; + + // validate inputs, all indices must be in the range [0, V) + for(int i = 0; i < B * T; i++) { + assert(0 <= inputs[i] && inputs[i] < V); + } + + // allocate space for all the activations if needed (done here, lazily) + if(model->acts_memory == NULL || B != model->batch_size || T != model->seq_len) { + // record the current B,T as well + model->batch_size = B; + model->seq_len = T; + // and now allocate the space + fill_in_activation_sizes(model->act_sizes, B, T, model->config); + size_t num_activations = 0; + for (size_t i = 0; i < NUM_ACTIVATION_TENSORS; i++) { + num_activations += model->act_sizes[i]; + } + model->num_activations = num_activations; + if (model->acts_memory != NULL) { + printf("GPT2 - Reallocating activations for B=%d T=%d\n", B, T); + cudaFree(model->acts_memory); + cudaFree(model->inputs); + } + model->acts_memory = malloc_and_point_activations(&model->acts, model->act_sizes); + printf("GPT2 - Allocated %zu MiB for activations\n", (num_activations * sizeof(float)) >> 20); // >> 20 is /(1024*1024) + cudaCheck(cudaMalloc((void**)&model->inputs, B * T * sizeof(int))); + cudaCheck(cudaMallocHost((void**)&model->cpu_losses, B * T * sizeof(float))); + } + + // copy inputs to the model + cudaCheck(cudaMemcpy(model->inputs, inputs, B * T * sizeof(int), cudaMemcpyHostToDevice)); + + // forward pass + ParameterTensors params = model->params; // for brevity + ActivationTensors acts = model->acts; + float* residual; + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe, B, T, C); // encoding goes into residual[0] + + for (int l = 0; l < L; l++) { + + residual = l == 0 ? acts.encoded : acts.residual3 + (l-1) * B * T * C; + + // get the pointers of the weights for this layer + float* l_ln1w = params.ln1w + l * C; + float* l_ln1b = params.ln1b + l * C; + float* l_qkvw = params.qkvw + l * 3*C * C; + float* l_qkvb = params.qkvb + l * 3*C; + float* l_attprojw = params.attprojw + l * C * C; + float* l_attprojb = params.attprojb + l * C; + float* l_ln2w = params.ln2w + l * C; + float* l_ln2b = params.ln2b + l * C; + float* l_fcw = params.fcw + l * 4*C * C; + float* l_fcb = params.fcb + l * 4*C; + float* l_fcprojw = params.fcprojw + l * C * 4*C; + float* l_fcprojb = params.fcprojb + l * C; + + // get the pointers of the activations for this layer + float* l_ln1 = acts.ln1 + l * B * T * C; + float* l_ln1_mean = acts.ln1_mean; + float* l_ln1_rstd = acts.ln1_rstd; + float* l_qkvr = acts.qkvr + l * B * T * 3*C; + float* l_atty = acts.atty + l * B * T * C; + float* l_att = acts.att + l * B * NH * T * T; + float* l_attproj = acts.attproj + l * B * T * C; + float* l_residual2 = acts.residual2 + l * B * T * C; + float* l_ln2 = acts.ln2 + l * B * T * C; + float* l_ln2_mean = acts.ln2_mean; + float* l_ln2_rstd = acts.ln2_rstd; + float* l_fch = acts.fch + l * B * T * 4*C; + float* l_fch_gelu = acts.fch_gelu + l * B * T * 4*C; + float* l_fcproj = acts.fcproj + l * B * T * C; + float* l_residual3 = acts.residual3 + l * B * T * C; + // these are only needed as scratchpads for the forward pass + float* scratch = acts.output; + + // now do the forward pass + layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C); + matmul_forward(scratch, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C); + attention_forward(l_atty, l_qkvr, l_att, scratch, B, T, C, NH); + matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C); + residual_forward(l_residual2, residual, l_attproj, B*T*C); + layernorm_forward(l_ln2, l_ln2_mean, l_ln2_rstd, l_residual2, l_ln2w, l_ln2b, B, T, C); + //op16 matmul + bias + GeLU fusion + // matmul_forward(l_fch, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + // gelu_forward(l_fch_gelu, l_fch, B*T*4*C); + matmul_forward_fused_gelu(l_fch_gelu, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + matmul_forward(l_fcproj, l_fch_gelu, l_fcprojw, l_fcprojb, B, T, 4*C, C); + residual_forward(l_residual3, l_residual2, l_fcproj, B*T*C); + } + + residual = acts.residual3 + (L-1) * B * T * C; // last residual is in residual3 + layernorm_forward(acts.lnf, acts.lnf_mean, acts.lnf_rstd, residual, params.lnfw, params.lnfb, B, T, C); + matmul_forward(acts.output, acts.lnf, params.wte, NULL, B, T, C, Vp); + + model->mean_loss = -1.0f; +} + +void gpt2_free(GPT2 *model) { + cudaCheck(cudaFree(model->params_memory)); + cudaCheck(cudaFree(model->grads_memory)); + cudaCheck(cudaFree(model->m_memory)); + cudaCheck(cudaFree(model->v_memory)); + cudaCheck(cudaFree(model->acts_memory)); + cudaCheck(cudaFree(model->grads_acts_memory)); + cudaCheck(cudaFree(model->inputs)); + cudaCheck(cudaFree(model->targets)); + cudaFreeHost(model->cpu_losses); +} + +#define GPT2_EOT 50256 \ No newline at end of file diff --git a/kernels_op_16/matmul.cuh b/kernels_op_16/matmul.cuh new file mode 100644 index 0000000..e14e440 --- /dev/null +++ b/kernels_op_16/matmul.cuh @@ -0,0 +1,73 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "cutlass/cutlass.h" +#include "cutlass/gemm/device/gemm.h" +#include "/work/hdd/bche/hshah6/cutlass/include/cutlass/epilogue/thread/linear_combination_gelu.h" + +//referenced: https://github.com/NVIDIA/cutlass/blob/main/examples/14_ampere_tf32_tensorop_gemm/ampere_tf32_tensorop_gemm.cu + +__global__ void bias_add_filler_kernel(float* __restrict__ out, const float* __restrict__ bias, int BT, int OC) { + int x = blockIdx.x * blockDim.x + threadIdx.x; + if (x < BT * OC) out[x] += bias[x % OC]; +} + +inline void matmul_forward(float* __restrict__ out, const float* __restrict__ inp, const float* __restrict__ weight, + const float* __restrict__ bias,int B,int T, int C, int OC) { + int BT =B*T; + float alpha = 1.0f; + float beta = 0.0f; + cublasSgemm(cublas_handle, CUBLAS_OP_T, CUBLAS_OP_N, OC, BT, C, &alpha, weight, C, inp, C, &beta, out, OC); + if(bias != NULL) { + int total = BT * OC; + bias_add_filler_kernel<<<(total + 255) / 256, 256>>>(out, bias, BT, OC); + } +} + +// Launch kernel here +inline void matmul_forward_fused_gelu(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + int BT = B*T; + + int total = BT * OC; + bias_add_filler_kernel<<<(total + 255) / 256, 256>>>(out, bias, BT, OC); + + using ElementOutput = float; + using ElementAccumulator = float; + using ElementCompute = float; + + using EpilogueOp = cutlass::epilogue::thread::LinearCombinationGELU::value, ElementAccumulator, ElementCompute>; + + using Gemm = cutlass::gemm::device::Gemm< + float, cutlass::layout::RowMajor, + float, cutlass::layout::ColumnMajor, + ElementOutput, cutlass::layout::RowMajor, + ElementAccumulator, + cutlass::arch::OpClassTensorOp, + cutlass::arch::Sm80, + cutlass::gemm::GemmShape<128, 128, 32>, + cutlass::gemm::GemmShape<64, 64, 32>, + cutlass::gemm::GemmShape<16, 8, 8>, + EpilogueOp, + cutlass::gemm::threadblock::GemmIdentityThreadblockSwizzle<>, + 2 // Stages + >; + + Gemm gemm_op; + + typename Gemm::Arguments args{ + {BT, OC, C}, // problem size + {inp, C}, // A matrix + {weight, C}, // B matrix + {out, OC}, // C matrix + {out, OC}, // D matrix (output) + {1.0f, 1.0f} // alpha, beta for the epilogue + }; + + cutlass::Status status = gemm_op(args); +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_op_7/sweep.py b/kernels_op_7/sweep.py new file mode 100644 index 0000000..862f041 --- /dev/null +++ b/kernels_op_7/sweep.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +import csv +import re +import subprocess +import sys +from itertools import product +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +BUILD = ROOT / "kernels_op_7" / "build" +CSV_PATH = ROOT / "kernels_op_7" / "sweep.csv" + +# Just BR x BC. 9 configs. BR=128 will OOM on smem (48KB cap), expected NOT_FOUND. +GRID = { + "br": [32, 64, 128], + "bc": [32, 64, 128], +} + +# from Makefile +NVCC = "nvcc" +CFLAGS = "-O3 -arch=sm_86 -std=c++17 -rdc=true -g -lineinfo" +INCLUDES = "-I/usr/local/cuda/include -I/work/hdd/bche/hshah6/cutlass/include" +LIBS = "-L/usr/local/cuda/lib64 -lcublas -lcublasLt" + +KERNEL_NAME = "flash_attention_forward_kernel" + + +def run(cmd): + return subprocess.run(cmd, shell=True, cwd=ROOT, capture_output=True, text=True) + + +def compile_one(tag, defs): + """Two-step compile matching the Makefile.""" + obj = BUILD / f"{tag}.o" + binary = BUILD / tag + r = run(f"{NVCC} {CFLAGS} {INCLUDES} {defs} -c test_gpt2.cu -o {obj}") + if r.returncode != 0: + return None, f"COMPILE FAILED:\n{r.stderr}" + r = run(f"{NVCC} {CFLAGS} -o {binary} {obj} {LIBS}") + if r.returncode != 0: + return None, f"LINK FAILED:\n{r.stderr}" + return binary, "" + + +def profile_one(binary, tag): + report = BUILD / tag + log = BUILD / f"{tag}.log" + cmd = ( + f"nsys profile --stats=true --force-overwrite=true " + f"--cuda-event-trace=false -o {report} {binary}" + ) + r = run(cmd) + log.write_text((r.stdout or "") + "\n" + (r.stderr or "")) + return log + + +def parse_avg_ns(log_path, kernel_name): + pattern = re.compile(rf"^\s*\d.*\b{re.escape(kernel_name)}\b") + for line in log_path.read_text().splitlines(): + if pattern.search(line): + cols = line.split() + if len(cols) >= 4: + return cols[3] + return "NOT_FOUND" + + +def main(): + BUILD.mkdir(parents=True, exist_ok=True) + configs = list(product(GRID["br"], GRID["bc"])) + + first_fail_printed = False + with CSV_PATH.open("w", newline="") as fp: + out = csv.writer(fp) + out.writerow(["BR", "BC", "fa2_avg_ns"]) + + for i, (br, bc) in enumerate(configs, 1): + tag = f"br{br}_bc{bc}" + + defs = f"-DSWEEP_BR={br} -DSWEEP_BC={bc}" + binary, err = compile_one(tag, defs) + + if binary is None: + if not first_fail_printed: + print(err) + first_fail_printed = True + out.writerow([br, bc, "BUILD_FAIL"]) + fp.flush() + continue + + log = profile_one(binary, tag) + out.writerow([br, bc, parse_avg_ns(log, KERNEL_NAME)]) + fp.flush() + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/kernels_op_7/sweep.slurm b/kernels_op_7/sweep.slurm new file mode 100644 index 0000000..47eb18b --- /dev/null +++ b/kernels_op_7/sweep.slurm @@ -0,0 +1,29 @@ +#!/bin/bash +#SBATCH --job-name="op7_sweep" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=4 +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --constraint="projects,perf,nvperf" +#SBATCH --account=bche-delta-gpu +#SBATCH --time=01:00:00 +#SBATCH --output=op7_sweep_%j.out +#SBATCH --mail-user=asing271@illinois.edu +#SBATCH --mail-type=BEGIN,END,FAIL + +module reset +module load cuda + +cd "$SLURM_SUBMIT_DIR" + +# 1. Save current kernels and swap in req_4. +BACKUP=$(mktemp -d) +cp kernels/*.cuh "$BACKUP/" +cp kernels_req_4/*.cuh kernels/ + +# 2. Always restore kernels/ on exit. +trap 'cp "$BACKUP"/*.cuh kernels/' EXIT +python3 kernels_op_7/sweep.py diff --git a/kernels_op_8/layernorm.cuh b/kernels_op_8/layernorm.cuh new file mode 100644 index 0000000..74a1c53 --- /dev/null +++ b/kernels_op_8/layernorm.cuh @@ -0,0 +1,58 @@ +#ifndef __LAYERNORM_KERNEL_CUH__ +#define __LAYERNORM_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +#define MAX_C 768 +__constant__ float d_weight[MAX_C]; +__constant__ float d_bias[MAX_C]; + +__global__ void layernorm_forward_kernel(float* __restrict__ out, float* __restrict__ mean, float* __restrict__ rstd, const float* __restrict__ inp, int B, int T, int C) { + // Implement this + int b = blockDim.x * blockIdx.x + threadIdx.x; + int t = blockDim.y * blockIdx.y + threadIdx.y; + + if (b >= B || t >= T) { + return; + } + + const float* __restrict__ x = inp + b * T * C + t * C; + float* __restrict__ o = out + b * T * C + t * C; + + float sum = 0.0f; + for (int c = 0; c < C; ++c) { + sum += x[c]; + } + float m = sum / C; + + sum = 0.0f; + for (int c = 0; c < C; ++c) { + float diff = x[c] - m; + sum += diff * diff; + } + float s = rsqrtf(sum / C + 1e-5f); + + for (int c = 0; c < C; ++c) { + float n = s * (x[c] - m); + o[c] = n * d_weight[c] + d_bias[c]; + } +} + +// Launch kernel here +void layernorm_forward(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + + cudaMemcpyToSymbolAsync(d_weight, weight, C * sizeof(float)); + cudaMemcpyToSymbolAsync(d_bias, bias, C * sizeof(float)); + + // Implement this + dim3 block(16,16); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y); + + layernorm_forward_kernel<<>>(out,mean, rstd, inp, B,T, C); +} + +#endif // __LAYERNORM_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_op_9/attention.cuh b/kernels_op_9/attention.cuh new file mode 100644 index 0000000..8a791cb --- /dev/null +++ b/kernels_op_9/attention.cuh @@ -0,0 +1,108 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* __restrict__ q, float* __restrict__ k, float* __restrict__ v, const float* __restrict__ inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(const float* __restrict__ inp, float* __restrict__ out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +__global__ void preattention_scores_kernel(const float* __restrict__ q, const float* __restrict__ k, int B, int T, int NH, int HS, float* __restrict__ preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B* T* NH* T){ + int b = (x / T / T / NH); + int nh = (x / T / T) % NH; + int t1 = (x / T) % T; + int t2 = x % T; + + float sum = 0.0f; + const float* __restrict__ q_ptr = q + b * NH * T * HS + nh * T * HS + t1 * HS; + const float* __restrict__ k_ptr = k + b * NH * T * HS + nh * T * HS + t2 * HS; + for (int i = 0; i < HS; i++) { + sum += q_ptr[i] * k_ptr[i]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } +} + +__global__ void attention_v_kernel(float* __restrict__ vaccum, const float* __restrict__ att, const float* __restrict__ v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + float sum = 0.0f; + const float* __restrict__ att_ptr = att + b * NH * T * T + nh * T * T + t * T; + for (int t2 = 0; t2 < T; t2++) { + sum += att_ptr[t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; + } + + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x-1) / block.x,1,1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_op_9/encoder.cuh b/kernels_op_9/encoder.cuh new file mode 100644 index 0000000..f7e118c --- /dev/null +++ b/kernels_op_9/encoder.cuh @@ -0,0 +1,33 @@ +#ifndef ENCODER_FORWARD_KERNEL_CUH +#define ENCODER_FORWARD_KERNEL_CUH + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void encoder_forward_kernel(float* __restrict__ out, const int* __restrict__ inp, const float* __restrict__ wte, const float* __restrict__ wpe, + int B, int T, int C) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + int y = blockDim.y * blockIdx.y + threadIdx.y; + + if(x < B && y < T){ + int token = inp[x*T + y]; + const float* __restrict__ wte_ptr = wte + token * C; + const float* __restrict__ wpe_ptr = wpe + y * C; + for(int i = 0; i < C; i ++){ + out[x*T*C + y*C + i] = wte_ptr[i] + wpe_ptr[i]; + } + } +} + +// Launch kernel here +void encoder_forward(float* out, const int* inp, const float* wte, const float* wpe, int B, int T, int C) { + // Implement this + dim3 block(16,16,1); + dim3 grid((B + block.x -1)/ block.x,(T+block.y-1)/block.y); + + encoder_forward_kernel<<>>(out, inp, wte, wpe,B,T,C); +} + + +#endif // ENCODER_FORWARD_KERNEL_CUH \ No newline at end of file diff --git a/kernels_op_9/gelu.cuh b/kernels_op_9/gelu.cuh new file mode 100644 index 0000000..01a8720 --- /dev/null +++ b/kernels_op_9/gelu.cuh @@ -0,0 +1,32 @@ +#ifndef GELU_KERNEL_CUH_ +#define GELU_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +#define GELU_SCALING_FACTOR sqrtf(2.0f / M_PI) + +__global__ void gelu_forward_kernel(float* __restrict__ out, const float* __restrict__ inp, int N) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + float val = inp[x]; + float cube = 0.044715f * val * val * val; + out[x] = 0.5f * val * (1.0f + tanhf(GELU_SCALING_FACTOR * (val + cube))); + } + +} + +// Launch kernel here +void gelu_forward(float* out, const float* inp, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x -1) / block.x); + + gelu_forward_kernel<<>>(out, inp, N); + +} + +#endif // GELU_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels_op_9/layernorm.cuh b/kernels_op_9/layernorm.cuh new file mode 100644 index 0000000..5b8b26e --- /dev/null +++ b/kernels_op_9/layernorm.cuh @@ -0,0 +1,51 @@ +#ifndef __LAYERNORM_KERNEL_CUH__ +#define __LAYERNORM_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void layernorm_forward_kernel(float* __restrict__ out, float* __restrict__ mean, float* __restrict__ rstd, const float* __restrict__ inp, const float* __restrict__ weight, + const float* __restrict__ bias, int B, int T, int C) { + // Implement this + int b = blockDim.x * blockIdx.x + threadIdx.x; + int t = blockDim.y * blockIdx.y + threadIdx.y; + + if (b >= B || t >= T) { + return; + } + + const float* __restrict__ x = inp + b * T * C + t * C; + float* __restrict__ o = out + b * T * C + t * C; + + float sum = 0.0f; + for (int c = 0; c < C; ++c) { + sum += x[c]; + } + float m = sum / C; + + sum = 0.0f; + for (int c = 0; c < C; ++c) { + float diff = x[c] - m; + sum += diff * diff; + } + float s = rsqrtf(sum / C + 1e-5f); + + for (int c = 0; c < C; ++c) { + float n = s * (x[c] - m); + o[c] = n * weight[c] + bias[c]; + } +} + +// Launch kernel here +void layernorm_forward(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + // Implement this + dim3 block(16,16); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y); + + layernorm_forward_kernel<<>>(out,mean, rstd, inp, weight, bias,B,T, C); +} + +#endif // __LAYERNORM_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_op_9/matmul.cuh b/kernels_op_9/matmul.cuh new file mode 100644 index 0000000..a47ff11 --- /dev/null +++ b/kernels_op_9/matmul.cuh @@ -0,0 +1,38 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void matmul_forward_kernel(float* __restrict__ out, const float* __restrict__ inp, const float* __restrict__ weight, + const float* __restrict__ bias,int B,int T, int C, int OC) { + // Implement this + int b = blockDim.x*blockIdx.x + threadIdx.x; // B + int t = blockDim.y*blockIdx.y + threadIdx.y; // T + int oc = blockDim.z*blockIdx.z + threadIdx.z; // OC + + if (b >= B || t >= T || oc >= OC) { + return; + } + + float sum = (bias != NULL) ? bias[oc] : 0.0f; + const float* __restrict__ inp_token_ptr = inp + b * T * C + t * C; + const float* __restrict__ w_r = weight + oc * C; + for(int i = 0; i < C; i++){ + sum += inp_token_ptr[i] * w_r[i] ; + } + out[b * T * OC + t * OC + oc] = sum; +} + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + dim3 block(8,8,8); + dim3 grid((B+block.x-1)/block.x, (T+block.y-1)/block.y, (OC+block.z-1)/block.z); + + matmul_forward_kernel<<>>(out, inp, weight, bias,B,T, C, OC); + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_op_9/residual.cuh b/kernels_op_9/residual.cuh new file mode 100644 index 0000000..b0b327f --- /dev/null +++ b/kernels_op_9/residual.cuh @@ -0,0 +1,25 @@ +#ifndef RESIDUAL_KERNEL_CUH_ +#define RESIDUAL_KERNEL_CUH_ + +#include +#include "../utils/cuda_utils.cuh" + +__global__ void residual_forward_kernel(float* __restrict__ out, const float* __restrict__ inp1, const float* __restrict__ inp2, int N) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N){ + out[x] = inp1[x] + inp2[x]; + } +} + +// Launch kernel here +void residual_forward(float* out, float* inp1, float* inp2, int N) { + // Implement this + dim3 block(32,1,1); + dim3 grid((N + block.x - 1)/ block.x); + + residual_forward_kernel<<>>(out, inp1, inp2, N); +} + +#endif // RESIDUAL_KERNEL_CUH_ \ No newline at end of file diff --git a/kernels_op_9/softmax.cuh b/kernels_op_9/softmax.cuh new file mode 100644 index 0000000..906d6cf --- /dev/null +++ b/kernels_op_9/softmax.cuh @@ -0,0 +1,37 @@ +#ifndef __SOFTMAX_KERNEL_CUH__ +#define __SOFTMAX_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void softmax_forward_kernel(float* __restrict__ out, float inv_temperature, const float* __restrict__ inp, int N, int T) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N*T){ + int own_pos = x % T; + const float* __restrict__ a = inp + x * T; + float* __restrict__ o = out + x * T; + + float maxval = -FLT_MAX; + for (int i = 0; i <= own_pos; i++) { + maxval = fmaxf(maxval,a[i]); + } + + float sumval = 0.0f; + for (int i = 0; i <= own_pos; ++i) { + float ev = expf(inv_temperature * (a[i] - maxval)); + sumval += ev; + o[i] = ev; + } + + float norm = 1.0f / sumval; + for (int i = 0; i <= own_pos; ++i) { + o[i] *= norm; + } + } +} + +#endif // __SOFTMAX_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_0/attention.cuh b/kernels_req_0/attention.cuh new file mode 100644 index 0000000..4d1dc64 --- /dev/null +++ b/kernels_req_0/attention.cuh @@ -0,0 +1,202 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +#define PREATT_T 128 +#define PREATT_U 8 +#define PREATT_S 16 + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + + // each block handles a (b,nh) head and a tile of (t1, t2) + int head = blockIdx.z; //whcih pair + int b = head / NH; + int nh = head % NH; + + int t1 = blockIdx.x * PREATT_T + threadIdx.x; // row of output query positin + int t2 = blockIdx.y * PREATT_U; //start column key position + + //head pointers + const float* q_head = q + (b * NH + nh) * T * HS; //[T,HS] + const float* k_head = k + (b * NH + nh) * T * HS; //[T,HS] + float* preatt_head = preatt + (b * NH +nh) * T * T; //[T,T] + + //reg tile for accumulation + float accumulation[PREATT_U] = {0.0f}; + + __shared__ float k_tile[PREATT_S][PREATT_U]; + + for(int tile_start = 0; tile_start < HS; tile_start+=PREATT_S){ + // shared loading + // K is [T, HS], K[t2..t2+U, tile_start..tile_start+S] + + int s_l = threadIdx.x / PREATT_U; + int u_l = threadIdx.x % PREATT_U; + + int hs_idx = tile_start + s_l; + int t2_idx = t2 + u_l; + + if (hs_idx < HS && t2_idx < T) { + k_tile[s_l][u_l] = k_head[t2_idx * HS + hs_idx]; + } else { + k_tile[s_l][u_l] = 0.0f; + } + + __syncthreads(); + + //each threads now loads its row of queries from global memory into registers and accumulates + if (t1 < T){ + #pragma unroll + for(int s = 0; s < PREATT_S; s++){ + int c = tile_start + s; + float q_val = (c < HS) ? q_head[t1 * HS +c] : 0.0f; + #pragma unroll + for(int u = 0; u < PREATT_U; u++){ + accumulation[u] += q_val * k_tile[s][u]; + } + } + } + __syncthreads(); + } + + if(t1 < T){ + #pragma unroll + for(int u = 0; u < PREATT_U; u ++){ + int t2_u = t2 + u; + if(t2_u < T){ + preatt_head[t1 * T + t2_u] = accumulation[u]; + } + } + } +} + +#define ATTV_T 128 +#define ATTV_U 8 +#define ATTV_S 16 + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int head = blockIdx.z; + int b = head / NH; + int nh = head % NH; + + int t = blockIdx.x * ATTV_T + threadIdx.x; // row of output query positin + int hs_base = blockIdx.y * ATTV_U; //start column key position + + const float* att_head = att + (b * NH + nh) * T * T; + const float* v_head = v + (b * NH + nh) * T * HS; + float* out_head = vaccum + (b * NH + nh) * T * HS; + + float accumulation[ATTV_U] = {0.0f}; + + __shared__ float v_tile[ATTV_S][ATTV_U]; + + for(int tile_start = 0; tile_start < T; tile_start += ATTV_S){ + int s_l = threadIdx.x / ATTV_U; + int u_l = threadIdx.x % ATTV_U; + + int t2_idx = tile_start + s_l; + int hs_idx = hs_base + u_l; + + if (t2_idx < T && hs_idx < HS) { + v_tile[s_l][u_l] = v_head[t2_idx * HS + hs_idx]; + } else { + v_tile[s_l][u_l] = 0.0f; + } + __syncthreads(); + + // Each thread loads att values from registers and accumulates + if (t < T) { + #pragma unroll + for (int s = 0; s < ATTV_S; s++) { + int t2 = tile_start + s; + float att_val = (t2 < T) ? att_head[t * T + t2] : 0.0f; + #pragma unroll + for (int u = 0; u < ATTV_U; u++) { + accumulation[u] += att_val * v_tile[s][u]; + } + } + } + __syncthreads(); + } + + if (t < T) { + #pragma unroll + for (int u = 0; u < ATTV_U; u++) { + int hs = hs_base + u; + if (hs < HS) { + out_head[t * HS + hs] = accumulation[u]; + } + } + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 block_preatt(PREATT_T); + dim3 grid_preatt((T + PREATT_T - 1) / PREATT_T,(T+PREATT_U-1)/PREATT_U,B*NH); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + dim3 block_attv(PREATT_T); + dim3 grid_attv((T + ATTV_T - 1) / ATTV_T,(HS+ATTV_U-1)/ATTV_U,B*NH); + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_0/matmul.cuh b/kernels_req_0/matmul.cuh new file mode 100644 index 0000000..95dacb4 --- /dev/null +++ b/kernels_req_0/matmul.cuh @@ -0,0 +1,90 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + +#define TILE_T 128 +#define TILE_U 8 +#define TILE_S 16 + +__global__ void matmul_forward_kernel(float* out, const float* inp, const float* weight, + const float* bias,int B,int T, int C, int OC) { + // Implement this + const int BT = B * T; + + // getting which pair this thread is going to process + const int row = blockIdx.x * TILE_T + threadIdx.x; + // OC + const int col = blockIdx.y * TILE_U; + + //register tile + float accumilation[TILE_U] = {0.0f}; + + //init the tile with bais + if(bias != NULL && row < BT){ + #pragma unroll + for(int i = 0; i < TILE_U; i++){ + if(col + i < OC){ + accumilation[i] = bias[col + i]; + } + } + } + + __shared__ float N_tile[TILE_S][TILE_U]; + + for(int tile_start = 0; tile_start < C; tile_start += TILE_S){ + const int s_l = threadIdx.x / TILE_U; + const int u_l = threadIdx.x % TILE_U; + + const int c_g = tile_start + s_l; + const int oc_g = col + u_l; + + if(c_g < C && oc_g < OC){ + N_tile[s_l][u_l] = weight[oc_g * C + c_g]; + } + else{ + N_tile[s_l][u_l] = 0.0f; + } + __syncthreads(); + + if(row < BT){ + #pragma unroll + for(int s = 0 ; s < TILE_S; s++){ + const int c = tile_start + s; + + float m = 0.0f; + if(c < C){ + m = inp[row * C + c] ; + } + #pragma unroll + for(int u = 0; u < TILE_U; u++){ + accumilation[u] +=m *N_tile[s][u]; + } + } + } + __syncthreads(); + } + if(row < BT){ + #pragma unroll + for(int u = 0; u < TILE_U; u++){ + if(col + u < OC){ + out[row*OC+col+u] = accumilation[u]; + } + } + } +} + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + const int BT = B*T; + dim3 block(TILE_T); + dim3 grid((BT+TILE_T-1)/TILE_T,(OC+TILE_U-1)/TILE_U); + + matmul_forward_kernel<<>>(out, inp, weight, bias,B,T, C, OC); + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_1/attention.cuh b/kernels_req_1/attention.cuh new file mode 100644 index 0000000..2d2eb7c --- /dev/null +++ b/kernels_req_1/attention.cuh @@ -0,0 +1,191 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" +#include + +using namespace nvcuda; + +#define WMMA_M 16 +#define WMMA_N 16 +#define WMMA_K 8 + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +#define PREATT_WARPS_M 2 +#define PREATT_WARPS_N 4 +#define PREATT_BM (PREATT_WARPS_M * WMMA_M) +#define PREATT_BN (PREATT_WARPS_N * WMMA_N) + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + // each block handles a (b,nh) head and a tile of (t1, t2) + int head = blockIdx.z; //whcih pair + int b = head / NH; + int nh = head % NH; + + //head pointers + const float* q_head = q + (b * NH + nh) * T * HS; //[T,HS] + const float* k_head = k + (b * NH + nh) * T * HS; //[T,HS] + float* preatt_head = preatt + (b * NH +nh) * T * T; //[T,T] + + int start_row = blockIdx.x * PREATT_BM; + int start_col = blockIdx.y * PREATT_BN; + + int warp_id = threadIdx.x / 32; + int warp_row = warp_id / PREATT_WARPS_N; + int warp_col = warp_id % PREATT_WARPS_N; + + int warp_out_row = start_row + warp_row * WMMA_M; + int warp_out_col = start_col + warp_col * WMMA_N; + + // Q * K^T: A=Q (row_major), B=K (col_major gives K^T) + wmma::fragment a_frag; + wmma::fragment b_frag; + wmma::fragment c_frag; + + wmma::fill_fragment(c_frag, 0.0f); + + for(int kk = 0; kk < HS; kk += WMMA_K){ + if(warp_out_row < T && kk < HS){ + wmma::load_matrix_sync(a_frag, q_head + warp_out_row * HS + kk, HS); + #pragma unroll + for(int i = 0; i < a_frag.num_elements; i++){ + a_frag.x[i] = wmma::__float_to_tf32(a_frag.x[i]); + } + } + + if(warp_out_col < T && kk < HS) { + wmma::load_matrix_sync(b_frag, k_head + warp_out_col * HS + kk, HS); + #pragma unroll + for (int i = 0; i < b_frag.num_elements; i++) + b_frag.x[i] = wmma::__float_to_tf32(b_frag.x[i]); + } + + wmma::mma_sync(c_frag, a_frag, b_frag, c_frag); + } + + //write to global mem + if(warp_out_row < T && warp_out_col < T){ + wmma::store_matrix_sync(preatt_head + warp_out_row * T + warp_out_col, c_frag, T, wmma::mem_row_major); + } +} + +#define ATTV_WARPS_M 2 +#define ATTV_WARPS_N 4 +#define ATTV_BM 32 +#define ATTV_BN 64 + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int head = blockIdx.z; //which pair + int b = head / NH; + int nh = head % NH; + + //head pointers + const float* att_head = att + (b * NH + nh) * T * T; //[T,HS] + const float* v_head = v + (b * NH + nh) * T * HS; //[T,HS] + float* out_head = vaccum + (b * NH +nh) * T * HS; //[T,T] + + int row_start = blockIdx.x * ATTV_BM; + int col_start = blockIdx.y * ATTV_BN; + + int warpId = threadIdx.x / 32; + int warp_row = warpId / ATTV_WARPS_N; + int warp_col = warpId % ATTV_WARPS_N; + + int warp_out_row = row_start + warp_row * WMMA_M; + int warp_out_col = col_start + warp_col * WMMA_N; + + // att * V: both row-major; standard matmul + wmma::fragment a_frag; + wmma::fragment b_frag; + wmma::fragment c_frag; + + wmma::fill_fragment(c_frag, 0.0f); + + for(int kk = 0; kk < T; kk+=WMMA_K){ + if (warp_out_row < T && kk < T) { + wmma::load_matrix_sync(a_frag, att_head + warp_out_row * T + kk, T); + #pragma unroll + for (int i = 0; i < a_frag.num_elements; i++) + a_frag.x[i] = wmma::__float_to_tf32(a_frag.x[i]); + } + + if (kk < T && warp_out_col < HS) { + wmma::load_matrix_sync(b_frag, v_head + kk * HS + warp_out_col, HS); + #pragma unroll + for (int i = 0; i < b_frag.num_elements; i++) + b_frag.x[i] = wmma::__float_to_tf32(b_frag.x[i]); + } + + wmma::mma_sync(c_frag, a_frag, b_frag, c_frag); + } + + if (warp_out_row < T && warp_out_col < HS) { + wmma::store_matrix_sync(out_head + warp_out_row * HS + warp_out_col, c_frag, HS, wmma::mem_row_major); + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 grid_preatt((T + PREATT_BM - 1) / PREATT_BM, (T + PREATT_BN - 1) / PREATT_BN, B * NH); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + dim3 grid_attv((T + ATTV_BM - 1) / ATTV_BM, (HS + ATTV_BN - 1) / ATTV_BN, B * NH); + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_1/matmul.cuh b/kernels_req_1/matmul.cuh new file mode 100644 index 0000000..98ae747 --- /dev/null +++ b/kernels_req_1/matmul.cuh @@ -0,0 +1,98 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include +#include "../utils/cuda_utils.cuh" + +using namespace nvcuda; + +#define WMMA_M 16 +#define WMMA_N 16 +#define WMMA_K 8 + +#define WARPS_M 4 +#define WARPS_N 2 +#define BM 64 +#define BN 32 +#define BK WMMA_K + +__global__ void matmul_forward_kernel(float* out, const float* inp, const float* weight, + const float* bias,int B,int T, int C, int OC) { + // Implement this + int BT = B*T; + + // block output + int start_row = blockIdx.x * BM; + int start_col = blockIdx.y * BN; + + // which warp + int warp_id = threadIdx.x / 32; + int warp_row = warp_id / WARPS_N; + int warp_col = warp_id % WARPS_N; + + int warp_out_row = start_row + warp_row * WMMA_M; + int warp_out_col = start_col + warp_col * WMMA_N; + + wmma::fragment a_frag; + wmma::fragment b_frag; + wmma::fragment c_frag; + + wmma::fill_fragment(c_frag, 0.0f); + + for(int k = 0; k < C; k += WMMA_K){ + if(warp_out_row < BT && k < C){ + wmma::load_matrix_sync(a_frag, inp + warp_out_row * C + k, C); + + #pragma unroll + for(int i = 0; i< a_frag.num_elements; i++){ + a_frag.x[i] = wmma::__float_to_tf32(a_frag.x[i]); + } + } + + if (warp_out_col < OC && k < C) { + wmma::load_matrix_sync(b_frag, weight + warp_out_col * C + k, C); + #pragma unroll + for (int i = 0; i < b_frag.num_elements; i++) { + b_frag.x[i] = wmma::__float_to_tf32(b_frag.x[i]); + } + } + + wmma::mma_sync(c_frag, a_frag, b_frag, c_frag); + } + + //store in s mem, add bais durinthe writw out for performance + __shared__ float out_tile[BM * BN]; + + int row_smem = warp_row * WMMA_M; + int col_smem = warp_col * WMMA_N; + wmma::store_matrix_sync(out_tile + row_smem * BN + col_smem, c_frag, BN, wmma::mem_row_major); + __syncthreads(); + + for(int i = threadIdx.x; i < BM*BN; i += blockDim.x){ + int r_l = i / BN; + int c_l = i % BN; + int r_g = start_row + r_l; + int c_g = start_col + c_l; + + if(r_g < BT && c_g < OC){ + if(bias != NULL){ + out_tile[i] += bias[c_g]; + } + out[r_g * OC + c_g] = out_tile[i]; + } + } +} + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + int BT = B*T; + dim3 block(256); + dim3 grid((BT + BM - 1) / BM, (OC + BN - 1) / BN); + matmul_forward_kernel<<>>(out, inp, weight, bias,B,T, C, OC); + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_2/attention.cuh b/kernels_req_2/attention.cuh new file mode 100644 index 0000000..343dba4 --- /dev/null +++ b/kernels_req_2/attention.cuh @@ -0,0 +1,128 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +// __global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ +// // int x = blockDim.x * blockIdx.x + threadIdx.x; + +// // if(x < B* T* NH* T){ +// // int b = (x / T / T / NH); +// // int nh = (x / T / T) % NH; +// // int t1 = (x / T) % T; +// // int t2 = x % T; + +// // float sum = 0.0f; +// // for (int i = 0; i < HS; i++) { +// // sum += q[b*NH*T*HS + nh*T*HS + t1*HS + i] * k[b*NH*T*HS + nh*T*HS + t2*HS + i]; +// // } +// // preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; +// // } +// DONT NEED +// } + +// __global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ +// // int x = blockDim.x * blockIdx.x + threadIdx.x; + +// // if(x < B*NH*T*HS){ +// // int b = (x / HS / T / NH); +// // int nh = (x / HS / T) % NH; +// // int t = (x / HS) % T; +// // int hs = x % HS; +// // float sum = 0.0f; +// // for (int t2 = 0; t2 < T; t2++) { +// // sum += att[b*NH*T*T + nh*T*T + t*T + t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; +// // } + +// // vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; +// // } + +// //DONT NEED +// } + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + float alpha = 1.0f; + float beta = 0.0f; + + for(int b = 0; b < B;b ++){ + for(int nh = 0; nh < NH; nh++){ + int head = b * NH + nh; + const float* q_head = q+ head * T * HS; + const float* k_head = k+ head * T * HS; + float* preatt_head = preatt + head * T * T; + + cublasSgemm(cublas_handle, CUBLAS_OP_T, CUBLAS_OP_N, T,T,HS, &alpha, k_head, HS, q_head, HS, &beta, preatt_head, T); + } + } + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + for(int b = 0; b < B;b ++){ + for(int nh = 0; nh < NH; nh++){ + int head = b * NH + nh; + const float* att_head = att + head * T * T; + const float* v_head = v + head * T * HS; + float* out_head = vaccum + head * T * HS; + + cublasSgemm(cublas_handle, CUBLAS_OP_N, CUBLAS_OP_N, HS,T,T, &alpha, v_head, HS, att_head, T, &beta, out_head, HS); + } + } + + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_2/matmul.cuh b/kernels_req_2/matmul.cuh new file mode 100644 index 0000000..db26d0a --- /dev/null +++ b/kernels_req_2/matmul.cuh @@ -0,0 +1,38 @@ +#ifndef __MATMUL_KERNEL_CUH__ +#define __MATMUL_KERNEL_CUH__ + +#include +#include "../utils/cuda_utils.cuh" + + +__global__ void bias_add_kernel (float* out, const float* bias, int BT, int OC){ + int x = blockIdx.x * blockDim.x + threadIdx.x; + if(x < BT*OC){ + out[x] += bias[x % OC]; + } +} + +// __global__ void matmul_forward_kernel(float* out, const float* inp, const float* weight, +// const float* bias,int B,int T, int C, int OC) { +// // no need for this kernel + +// } + +// Launch kernel here +void matmul_forward(float* out, const float* inp, const float* weight, const float* bias, + int B, int T, int C, int OC) { + // Implement this + int BT = B*T; + float alpha = 1.0f; + float beta = 0.0f; + + cublasSgemm(cublas_handle, CUBLAS_OP_T, CUBLAS_OP_N, OC, BT, C, &alpha, weight, C, inp, C, &beta, out, OC); + + if (bias != NULL) { + int total = BT * OC; + bias_add_kernel<<<(total + 255) / 256, 256>>>(out, bias, BT, OC); + } + +} + +#endif // __MATMUL_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_3/attention.cuh b/kernels_req_3/attention.cuh new file mode 100644 index 0000000..51ed287 --- /dev/null +++ b/kernels_req_3/attention.cuh @@ -0,0 +1,105 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B* T* NH* T){ + int b = (x / T / T / NH); + int nh = (x / T / T) % NH; + int t1 = (x / T) % T; + int t2 = x % T; + + float sum = 0.0f; + for (int i = 0; i < HS; i++) { + sum += q[b*NH*T*HS + nh*T*HS + t1*HS + i] * k[b*NH*T*HS + nh*T*HS + t2*HS + i]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } +} + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + float sum = 0.0f; + for (int t2 = 0; t2 < T; t2++) { + sum += att[b*NH*T*T + nh*T*T + t*T + t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; + } + + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } +} + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(32,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x-1) / block.x,1,1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0 / sqrtf(HS); + int softmax_rows = B * NH * T; + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_3/layernorm.cuh b/kernels_req_3/layernorm.cuh new file mode 100644 index 0000000..49cd28b --- /dev/null +++ b/kernels_req_3/layernorm.cuh @@ -0,0 +1,102 @@ +#ifndef __LAYERNORM_KERNEL_CUH__ +#define __LAYERNORM_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +#define BLOCK_DIM 256 +#define WARP_SIZE 32 + +//lecture 15 slide ~32 +__device__ __inline__ float warp_reduce(float val) { + float partialSum = val; + for(unsigned int stride = WARP_SIZE/2; stride > 0; stride /= 2) { + partialSum += __shfl_down_sync(0xffffffff, partialSum, stride); + } + return partialSum; +} + +__global__ void layernorm_forward_kernel(float* out, float* mean, float* rstd, const float* inp, const float* weight, + const float* bias, int B, int T, int C) { + // Implement this + int bt = blockIdx.x; + if(bt >= B*T){ + return; + } + + const float* x = inp + bt * C; + float* o = out + bt * C; + + // computer mean + float partial_sum = 0.0f; + for(int c = threadIdx.x ; c < C ; c += BLOCK_DIM){ + partial_sum += x[c]; + } + + partial_sum = warp_reduce(partial_sum); + + __shared__ float partial_sum_s[BLOCK_DIM / WARP_SIZE]; + if(threadIdx.x % WARP_SIZE == 0){ + partial_sum_s[threadIdx.x / WARP_SIZE] = partial_sum; + } + + __syncthreads(); + + if(threadIdx.x < WARP_SIZE){ + partial_sum = (threadIdx.x < BLOCK_DIM / WARP_SIZE) ? partial_sum_s[threadIdx.x] : 0.0f; + partial_sum = warp_reduce(partial_sum); + } + + //thread 0 will have the final sum + __shared__ float s_mean; + if (threadIdx.x == 0) { + s_mean = partial_sum / C; + } + + __syncthreads(); + float m = s_mean; + + //compute variance + partial_sum = 0.0f; + for (int c = threadIdx.x; c < C; c += BLOCK_DIM) { + float diff = x[c] - m; + partial_sum += diff * diff; + } + + partial_sum = warp_reduce(partial_sum); + + if (threadIdx.x % WARP_SIZE == 0) { + partial_sum_s[threadIdx.x / WARP_SIZE] = partial_sum; + } + __syncthreads(); + + if (threadIdx.x < WARP_SIZE) { + partial_sum = (threadIdx.x < BLOCK_DIM / WARP_SIZE) ? partial_sum_s[threadIdx.x] : 0.0f; + partial_sum = warp_reduce(partial_sum); + } + + __shared__ float s_rstd; + if (threadIdx.x == 0) { + s_rstd = rsqrtf(partial_sum / C + 1e-5f); + } + __syncthreads(); + float s = s_rstd; + + //normalize + for (int c = threadIdx.x; c < C; c += BLOCK_DIM) { + float n = s * (x[c] - m); + o[c] = n * weight[c] + bias[c]; + } +} + +// Launch kernel here +void layernorm_forward(float* out, float* mean, float* rstd, float* inp, float* weight, float* bias, + int B, int T, int C) { + // Implement this + int num_blocks = B*T; + layernorm_forward_kernel<<>>(out,mean, rstd, inp, weight, bias,B,T, C); +} + +#endif // __LAYERNORM_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_3/softmax.cuh b/kernels_req_3/softmax.cuh new file mode 100644 index 0000000..52a5c6a --- /dev/null +++ b/kernels_req_3/softmax.cuh @@ -0,0 +1,99 @@ +#ifndef __SOFTMAX_KERNEL_CUH__ +#define __SOFTMAX_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +#define BLOCK_DIM 256 +#define WARP_SIZE 32 + +// lecture 15 slide 32 +__device__ __inline__ float warpReduceSum(float partialSum) { + for (unsigned int stride = WARP_SIZE / 2; stride > 0; stride /= 2) { + partialSum += __shfl_down_sync(0xffffffff, partialSum, stride); + } + return partialSum; +} + +// same pattern for max +__device__ __inline__ float warpReduceMax(float partialMax) { + for (unsigned int stride = WARP_SIZE / 2; stride > 0; stride /= 2) { + partialMax = fmaxf(partialMax, __shfl_down_sync(0xffffffff, partialMax, stride)); + } + return partialMax; +} + +__global__ void softmax_forward_kernel(float* out, float inv_temperature, const float* inp, int N, int T) { + // Implement this + int row = blockIdx.x; + if(row >= N*T){ + return; + } + + int own_pos = row % T; + const float* a = inp + row * T; + float* o = out + row * T; + + //find max + //thread coarsening + float partial_max = -FLT_MAX; + for(int i = threadIdx.x; i <= own_pos; i+=BLOCK_DIM){ + partial_max = fmaxf(partial_max,a[i]); + } + + //lecture 15 slide 38 + partial_max = warpReduceMax(partial_max); + + __shared__ float partial_max_s[BLOCK_DIM / WARP_SIZE]; + if (threadIdx.x % WARP_SIZE == 0) { + partial_max_s[threadIdx.x / WARP_SIZE] = partial_max; + } + __syncthreads(); + + if (threadIdx.x < WARP_SIZE) { + partial_max = (threadIdx.x < BLOCK_DIM / WARP_SIZE) ? partial_max_s[threadIdx.x] : -FLT_MAX; + partial_max = warpReduceMax(partial_max); + } + + __shared__ float s_max; + if (threadIdx.x == 0) { + s_max = partial_max; + } + __syncthreads(); + float maxval = s_max; + + float partial_sum = 0.0f; + for (int i = threadIdx.x; i <= own_pos; i += BLOCK_DIM) { + float ev = expf(inv_temperature * (a[i] - maxval)); + o[i] = ev; + partial_sum += ev; + } + + partial_sum = warpReduceSum(partial_sum); + + __shared__ float partial_sums_s[BLOCK_DIM / WARP_SIZE]; + if (threadIdx.x % WARP_SIZE == 0) { + partial_sums_s[threadIdx.x / WARP_SIZE] = partial_sum; + } + __syncthreads(); + + if (threadIdx.x < WARP_SIZE) { + partial_sum = (threadIdx.x < BLOCK_DIM / WARP_SIZE) ? partial_sums_s[threadIdx.x] : 0.0f; + partial_sum = warpReduceSum(partial_sum); + } + + __shared__ float s_sum; + if (threadIdx.x == 0) { + s_sum = partial_sum; + } + __syncthreads(); + float norm = 1.0f / s_sum; + + for (int i = threadIdx.x; i <= own_pos; i += BLOCK_DIM) { + o[i] *= norm; + } +} + +#endif // __SOFTMAX_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_4/attention.cuh b/kernels_req_4/attention.cuh new file mode 100644 index 0000000..c10f16d --- /dev/null +++ b/kernels_req_4/attention.cuh @@ -0,0 +1,259 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + + +//added templates so the compiler can unroll loops and optimize better. +template +__global__ void flash_attention_forward_kernel(const float* __restrict__ Q, const float* __restrict__ K, const float* __restrict__ V, float* __restrict__ out, int B, int T, int NH, float scale) { + //following line by like psudo code from flash attention paper. + // https://tridao.me/publications/flash2/flash2.pdf + + // step 1 map thread blocks to (batch , head , query_tile) + int b = blockIdx.x; + int nh = blockIdx.y; + int query_tile = blockIdx.z; + + int query_row = threadIdx.x; //load index + int query_global = query_tile * BR + query_row; + + //step 2 calculate all the base pointers for this block. We will be loading queries from Q and keys/values from K and V, and writing output to out. + const float* __restrict__ query_base = Q + b * NH * T * HS + nh * T * HS; + const float* __restrict__ key_base = K + b * NH * T * HS + nh * T * HS; + const float* __restrict__ value_base = V + b * NH * T * HS + nh * T * HS; + float* out_base = out + b * NH * T * HS + nh * T * HS; + + // declare shared memory for the tile of queries and keys/values. We will load a tile of queries and then iterate over tiles of keys/values. + __shared__ float smem_q[BR * HS]; + __shared__ float smem_k[BC * HS]; + __shared__ float smem_v[BC * HS]; + + // we will compute the output for a tile of queries at a time. + int query_tile_start = query_tile * BR; // global index of the first query in this tile. This is used to load the correct queries into shared memory and also to know which output rows we are responsible for computing. + bool q_valid = (query_global < T); // check if this thread's query is within the sequence length. If not, it will do nothing. + + // step 4 load query tiles from global memory to shared memory. + // coalesced load of the queries for this tile into shared memory. a linear load of BR*HS elements, with each thread loading HS elements for one query. + #pragma unroll + for (int i = 0; i < HS; i++) { + smem_q[query_row * HS + i] = (query_tile_start + query_row < T) ? query_base[(query_tile_start + query_row) * HS + i] : 0.0f; // if the query is out of bounds, we can load zeros since they won't contribute to the output. + } + + // step 5 init accumulators for the output of this tile of queries. + float out_reg[HS]; + #pragma unroll + for (int i = 0; i < HS; i++) { + out_reg[i] = 0.0f; + } + float max_score = -FLT_MAX; + float sum_exp = 0.0f; + + + // step 6 determine KV tile range for casual masking + int query_tile_end = (query_tile + 1) * BR; + int n_kv_tiles = (query_tile_end < T) ? (T + BC - 1) / BC : (query_tile_end + BC - 1) / BC; // if the end of the query tile is within the sequence length, we only need to iterate over the KV tiles that cover the sequence length. If the end of the query tile exceeds the sequence length, we need to iterate over all KV tiles that cover the end of the query tile to properly apply casual masking. + + __syncthreads(); + + //step 7 main loop over kv values + for(int j = 0; j < n_kv_tiles; j++){ + __syncthreads(); + + int k_tile_start = j * BC; + + // step 8 load k and v tiles from global memory + #pragma unroll + for (int i = 0; i < HS; i++) { + int l = i * BC + query_row; + int row = l / HS; + int col = l % HS; + int global_row = k_tile_start + row; + bool valid = (global_row < T) && (row < BC); + + if (valid) { + smem_k[row * HS + col] = key_base[global_row * HS + col]; + smem_v[row * HS + col] = value_base[global_row * HS + col]; + } else { + smem_k[row * HS + col] = 0.0f; // pad with zeros if out of bounds + smem_v[row * HS + col] = 0.0f; // pad with zeros if out of bounds + } + } + __syncthreads(); + + // step 9 compute one row of S = Qi * Kj^T and apply casual masking + float score_reg[BC]; + float row_max = -FLT_MAX; + + #pragma unroll + for (int c = 0; c < BC; c++) { + int k_global = j * BC + c; + float dot; // compute the dot product of the query and key for this position + + if (!q_valid || k_global > query_global) { // apply casual masking. If the key position is greater than the query position, we mask it out by setting the score to a very large negative number. + dot = -FLT_MAX; + } else { + float acc = 0.0f; + #pragma unroll + for (int h = 0; h < HS; h++) { + acc += smem_q[query_row * HS + h] * smem_k[c * HS + h]; + } + dot = acc * scale; + } + score_reg[c] = dot; + row_max = fmaxf(row_max, dot); // local max for this row + } + + // step 10 online softmax rescaling + float max_new = fmaxf(max_score, row_max); + if (max_new == -FLT_MAX) { // if both are -FLT_MAX, we can skip the rescaling because all scores are -FLT_MAX and the output will be zero anyway. + continue; + } + float alpha = (max_score == -FLT_MAX) ? 0.0f : expf(max_score - max_new); // if max_score is -FLT_MAX, we set alpha to 0 to avoid exp overflow. Otherwise, we compute the scaling factor for the old scores. + float local_tile = 0.0f; // this will hold the sum of exp for the scores in this tile after rescaling. + + #pragma unroll + for (int c = 0; c < BC; c++) { + float p = (score_reg[c] == -FLT_MAX) ? 0.0f : expf(score_reg[c] - max_new); // compute the exp for this score after rescaling. If the score is -FLT_MAX, we set it to 0 to avoid exp underflow. + local_tile += p; + score_reg[c] = p; // store the rescaled exp in score_reg for later use in the weighted sum. + } + + // update output acc + #pragma unroll + for (int h = 0; h < HS; ++h) { + float pv = 0.0f; + #pragma unroll + for (int c = 0; c < BC; ++c) { + pv += score_reg[c] * smem_v[c * HS + h]; + } + out_reg[h] = alpha * out_reg[h] + pv; + } + sum_exp = alpha * sum_exp + local_tile; // update the local tile sum with the contribution from this tile and the previous tiles, scaled by alpha. + max_score = max_new; // update the max score for the next iteration + + } + if (q_valid) { + float inv_l = (sum_exp > 0.0f) ? (1.0f / sum_exp) : 0.0f; + #pragma unroll + for (int h = 0; h < HS; ++h) { + out_base[query_global * HS + h] = out_reg[h] * inv_l; + } + } +} + + + +__global__ void permute_kernel(float* __restrict__ q, float* __restrict__ k, float* __restrict__ v, const float* __restrict__ inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(const float* __restrict__ inp, float* __restrict__ out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +// __global__ void preattention_scores_kernel(const float* __restrict__ q, const float* __restrict__ k, int B, int T, int NH, int HS, float* __restrict__ preatt){ +// int x = blockDim.x * blockIdx.x + threadIdx.x; + +// if(x < B* T* NH* T){ +// int b = (x / T / T / NH); +// int nh = (x / T / T) % NH; +// int t1 = (x / T) % T; +// int t2 = x % T; + +// float sum = 0.0f; +// const float* __restrict__ q_ptr = q + b * NH * T * HS + nh * T * HS + t1 * HS; +// const float* __restrict__ k_ptr = k + b * NH * T * HS + nh * T * HS + t2 * HS; +// for (int i = 0; i < HS; i++) { +// sum += q_ptr[i] * k_ptr[i]; +// } +// preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; +// } +// } + +// __global__ void attention_v_kernel(float* __restrict__ vaccum, const float* __restrict__ att, const float* __restrict__ v, int B, int T, int NH, int HS){ +// int x = blockDim.x * blockIdx.x + threadIdx.x; + +// if(x < B*NH*T*HS){ +// int b = (x / HS / T / NH); +// int nh = (x / HS / T) % NH; +// int t = (x / HS) % T; +// int hs = x % HS; +// float sum = 0.0f; +// const float* __restrict__ att_ptr = att + b * NH * T * T + nh * T * T + t * T; +// for (int t2 = 0; t2 < T; t2++) { +// sum += att_ptr[t2] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; +// } + +// vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; +// } +// } + +// Launch all kernels related to attention here +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + // Implement this + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(256,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + //fa2 — tile sizes overridable via -DSWEEP_BR=... -DSWEEP_BC=... for op_7 sweep +#ifdef SWEEP_BR + constexpr int BR = SWEEP_BR; +#else + constexpr int BR = 64; +#endif +#ifdef SWEEP_BC + constexpr int BC = SWEEP_BC; +#else + constexpr int BC = 64; +#endif + float scale = 1.0f / sqrtf((float)HS); + + int number_of_q_tiles = (T + BR - 1) / BR; + dim3 fa_grid(B, NH, number_of_q_tiles); // one block per (batch, head, Q tile) + dim3 fa_block(BR, 1, 1); // one warp per query in the tile + + float* vaccum = inp; + flash_attention_forward_kernel<<>>(q, k, v, vaccum, B, T, NH, scale); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_5/local_attention.cuh b/kernels_req_5/local_attention.cuh new file mode 100644 index 0000000..b254d78 --- /dev/null +++ b/kernels_req_5/local_attention.cuh @@ -0,0 +1,138 @@ +#ifndef __ATTENTION_CUH__ +#define __ATTENTION_CUH__ + +// WINDOW = 129 because the CPU reference uses an inclusive range [q_t - 128, q_t], +// which covers 129 positions (128 look-back steps + the current token itself). +#define WINDOW 129 + +#include +#include "../utils/cuda_utils.cuh" +#include "local_softmax.cuh" +#include + +__global__ void permute_kernel(float* q, float* k, float* v, const float* inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(float* inp, float *out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +__global__ void preattention_scores_kernel(float* q, float* k, int B, int T, int NH, int HS, float* preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + //Make int terms of WINDOW instead of full T, so its T * WINDOW + if(x < B* T* NH* WINDOW){ + int b = (x / WINDOW / T / NH); + int nh = (x / WINDOW / T) % NH; + int t1 = (x / WINDOW) % T; + int w = x % WINDOW; + + //Key start point based on offset + int t2 = t1 - w; + + float sum = 0.0f; + + //Need to check to not look too far back and stay in window range + //First check if t2 (Key Step) is behind or equal to t1 (Query Step) + // int lowest_margin = t1 - WINDOW; + if (t2 >= 0) { + for (int i = 0; i < HS; i++) { + sum += q[b*NH*T*HS + nh*T*HS + t1*HS + i] * k[b*NH*T*HS + nh*T*HS + t2*HS + i]; + } + } + else { + sum = -FLT_MAX; + } + preatt[b * NH * T * WINDOW + nh * T * WINDOW + t1 * WINDOW + w] = sum; + //preatt[x] = sum; + } +} + +__global__ void attention_v_kernel(float* vaccum, float* att, float* v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t1 = (x / HS) % T; + int hs = x % HS; + float sum = 0.0f; + for (int w = 0; w < WINDOW; w++) { + int t2 = t1 - w; + if (t2 >= 0) { + sum += att[b*NH*T*WINDOW + nh*T*WINDOW + t1*WINDOW + w] * v[b*NH*T*HS + nh*T*HS + t2*HS + hs]; + } + } + + vaccum[b * NH * T * HS + nh * T * HS + t1 * HS + hs] = sum; + //vaccum[x] = sum; + } +} + +// Launch all kernels related to attention here +void local_attention_forward_gpu(float* out, float* qkvr, float* att, float* inp, int B, int T, int NH, int HS) { + // Implement this + + float *q, *k, *v; + q = qkvr + 0 * B * T * NH * HS; + k = qkvr + 1 * B * T * NH * HS; + v = qkvr + 2 * B * T * NH * HS; + + dim3 block(256,1,1); + dim3 grid((B * NH * T * HS + block.x-1) / block.x,1,1); + + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + float* preatt; + cudaMalloc(&preatt, B * NH * T * WINDOW * sizeof(float)); + + dim3 grid_preatt((B * NH * T * WINDOW + block.x-1) / block.x,1,1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float* packed_att; + cudaMalloc(&packed_att, B * NH * T * WINDOW * sizeof(float)); + + float scale = 1.0 / sqrtf(HS); + dim3 grid_softmax (( B * NH * T + block.x -1 )/ block.x,1,1); + local_softmax_forward_kernel<<>>(packed_att, scale, preatt, B * NH, T); + + float* vaccum; + cudaMalloc(&vaccum, B * NH * T * HS * sizeof(float)); + + attention_v_kernel<<>>(vaccum, packed_att, v, B, T, NH, HS); + + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); + + cudaFree(preatt); + cudaFree(vaccum); + cudaFree(packed_att); +} + +#endif // __ATTENTION_CUH__ \ No newline at end of file diff --git a/kernels_req_5/local_softmax.cuh b/kernels_req_5/local_softmax.cuh new file mode 100644 index 0000000..62672a1 --- /dev/null +++ b/kernels_req_5/local_softmax.cuh @@ -0,0 +1,37 @@ +#ifndef __SOFTMAX_KERNEL_CUH__ +#define __SOFTMAX_KERNEL_CUH__ + +#include +#include +#include +#include "../utils/cuda_utils.cuh" + +__global__ void local_softmax_forward_kernel(float* out, float inv_temperature, const float* inp, int N, int T) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < N*T){ + //row offset based on WIndow size + int rowoff = x * WINDOW; + + float maxval = -FLT_MAX; + for (int i = 0; i < WINDOW; i++) { + maxval = fmaxf(maxval, inp[i + rowoff]); + } + + float sumval = 0.0f; + //Traverse Window Length Again + for (int i = 0; i < WINDOW; i++) { + float ev = expf(inv_temperature * (inp[i + rowoff] - maxval)); + sumval += ev; + out[i + rowoff] = ev; + } + + float norm = 1.0f / sumval; + for (int i = 0; i < WINDOW; i++) { + out[i + rowoff] *= norm; + } + } +} + +#endif // __SOFTMAX_KERNEL_CUH__ \ No newline at end of file diff --git a/kernels_req_6/attention.cuh b/kernels_req_6/attention.cuh new file mode 100644 index 0000000..0e018e1 --- /dev/null +++ b/kernels_req_6/attention.cuh @@ -0,0 +1,309 @@ +#ifndef __ATTENTION_KV_CACHE_TEMPLATE_CUH__ +#define __ATTENTION_KV_CACHE_TEMPLATE_CUH__ + +#include +#include "../utils/cuda_utils.cuh" +#include "softmax.cuh" +#include +#include + +extern bool g_enable_kv_cache; // Toggles KV-caching logic +extern float* g_kv_cache; // Pointer to the GPU-allocated KV cache buffer +extern int g_layer_idx; // Tracks the current transformer layer, useful for knowing when to switch to decode +extern bool g_is_prefill; // True during the initial prompt, False during token generation +extern int g_current_pos; // Tracks the absolute sequence position during Decode, helps index into the KV cache +constexpr int MAX_SEQ_LEN = 1024; // gpt 2 default +// --- Suggested Kernel Placeholders (TODO: Add your regular attention kernel here, and any other kernels you may need) --- +//all the tests are running out of mem to store KV so implementing dynamic alloc +static int s_cache_b = 0; +static int s_cache_nh = 0; +static int s_cache_hs = 0; + + +__global__ void permute_kernel(float* __restrict__ q, float* __restrict__ k, float* __restrict__ v, const float* __restrict__ inp, int B, int N, int NH, int d) { + // Implement this + + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + int inp_idx = (b * N * 3 * NH * d) + (t * 3 * NH * d) + (0 * NH * d) + (nh * d) + d_; + q[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx]; + k[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + NH * d]; + v[b * NH * N * d + nh * N * d + t * d + d_] = inp[inp_idx + 2 * (NH * d)]; + } +} + + +__global__ void unpermute_kernel(const float* __restrict__ inp, float* __restrict__ out, int B, int N, int NH, int d) { + // Implement this + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if( x < B*NH*N*d){ + int b = (x / d / N / NH); + int nh = (x / d / N) % NH; + int t = (x / d) % N; + int d_ = x % d; + + out[b*N*NH*d + t*NH*d + nh*d + d_] = inp[b*NH*N*d + nh*N*d + t*d + d_]; + } + +} + +// base line kernels + +__global__ void preattention_scores_kernel(const float* __restrict__ q, const float* __restrict__ k, int B, int T, int NH, int HS, float* __restrict__ preatt){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B* T* NH* T){ + int b = (x / T / T / NH); + int nh = (x / T / T) % NH; + int t1 = (x / T) % T; + int t2 = x % T; + + float sum = 0.0f; + const float* __restrict__ q_ptr = q + b * NH * T * HS + nh * T * HS + t1 * HS; + const float* __restrict__ k_ptr = k + b * NH * T * HS + nh * T * HS + t2 * HS; + for (int i = 0; i < HS; i++) { + sum += q_ptr[i] * k_ptr[i]; + } + preatt[b * NH * T * T + nh * T * T + t1 * T + t2] = sum; + } +} + +__global__ void attention_v_kernel(float* __restrict__ vaccum, const float* __restrict__ att, const float* __restrict__ v, int B, int T, int NH, int HS){ + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B*NH*T*HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + + float sum = 0.0f; + const float* __restrict__ att_ptr = att + b * NH * T * T + nh * T * T + t * T; + const float* __restrict__ v_ptr = v + b * NH * T * HS + nh * T * HS; + for (int i = 0; i < T; i++) { + sum += att_ptr[i] * v_ptr[i * HS + hs]; + } + vaccum[b * NH * T * HS + nh * T * HS + t * HS + hs] = sum; + } +} + + + +// helper kernel to calculate index into 1D cache buffer +__device__ inline int get_cache_index(int layer, int is_v, int b, int nh, int t, int B, int NH, int HS) { + // our layout layer -> k/v -> batch -> head -> time -> head_size + return ((((layer * 2 + is_v) * B + b) * NH + nh) * MAX_SEQ_LEN + t) * HS; +} + + +/** + * Kernel to store the newly computed Key and Value into the global KV cache. + * CAN BE FUSED WITH QKV MATMUL + * In Prefill: Stores T tokens for the current layer. + * In Decode: Stores 1 token at g_current_pos for the current layer. + */ +__global__ void store_kv_kernel(float* cache, const float* k, const float* v,int layer, int B, int NH, int HS, int T, int current_pos, bool is_prefill) { + // Implement logic to copy K and V from the input QKV buffer into g_kv_cache + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x < B * NH * T * HS){ + int b = (x / HS / T / NH); + int nh = (x / HS / T) % NH; + int t = (x / HS) % T; + int hs = x % HS; + + //check where to write in the cache + int cache_idx; + if(is_prefill){ + cache_idx = t; + } else { + cache_idx = current_pos; + } + + // Write K and V to the cache + int cache_source_idx = b * NH * T * HS + nh * T * HS + t * HS + hs; + cache[get_cache_index(layer, 0, b, nh, cache_idx, B, NH, HS) + hs] = k[cache_source_idx]; // is_v = 0 for K + cache[get_cache_index(layer, 1, b, nh, cache_idx, B, NH, HS) + hs] = v[cache_source_idx]; // is_v = 1 for V + } +} + +/** + * Kernel to compute the attention for the Decode phase (T=1). + * This involves: + * 1. Extracting the current Query (Q) + * 2. Gathering all previous Keys (K) and Values (V) from the cache + * 3. Computing attention using the new Q and cached KV values. + */ +__global__ void decode_attention_kernel(float* vaccum, const float* q, const float* cache, int layer, int B, int NH, int HS, int current_pos) { + // Implement T=1 attention logic using the KV cache + int x = blockDim.x * blockIdx.x + threadIdx.x; + + if(x >= B*NH*HS){ + return; + } + + int b = (x / HS / NH); + int nh = (x / HS) % NH; + int hs = x % HS; + + float max_val = -FLT_MAX; + float scale = rsqrtf((float)HS); + const float* __restrict__ q_ptr = q + b * NH * HS + nh * HS; + + // Q * K^T for all previous tokens in the cache + for(int t = 0; t <= current_pos; t++){ + const float* __restrict__ k_ptr = cache + get_cache_index(layer, 0, b, nh, t, B, NH, HS); // is_v = 0 for K + float score = 0.0f; + for(int i = 0; i < HS; i++){ + score += q_ptr[i] * k_ptr[i]; + } + score *= scale; + if(score > max_val){ + max_val = score; + } + } + + // Compute softmax + float sum_exp = 0.0f; + float output = 0.0f; + + for(int t = 0; t <= current_pos; t++){ + const float* __restrict__ k_ptr = cache + get_cache_index(layer, 0, b, nh, t, B, NH, HS); + float score = 0.0f; + for(int i = 0; i < HS; i++){ + score += q_ptr[i] * k_ptr[i]; + } + score *= scale; + + float exp_score = expf(score - max_val); + sum_exp += exp_score; + + const float* __restrict__ v_ptr = cache + get_cache_index(layer, 1, b, nh, t, B, NH, HS); + output += exp_score * v_ptr[hs]; + } + vaccum[b * NH * HS + nh * HS + hs] = output / sum_exp; +} + +//alloc and realloc kv cache +static void allocate_kv_cache(int B, int NH, int HS){ + if(g_kv_cache != nullptr && s_cache_b == B && s_cache_nh == NH && s_cache_hs == HS){ + // If the cache exists and the dimensions match, do nothing + return; + } + + // If the cache exists but the dimensions don't match, free and reallocate + if(g_kv_cache != nullptr){ + cudaFree(g_kv_cache); + g_kv_cache = nullptr; + } + + size_t total_cache_size = 12 * 2 * B * NH * MAX_SEQ_LEN * HS * sizeof(float); // 12 layers, K and V, max seq len + cudaMalloc(&g_kv_cache, total_cache_size); + cudaMemset(g_kv_cache, 0, total_cache_size); // 0 the cache + + s_cache_b = B; + s_cache_nh = NH; + s_cache_hs = HS; +} + + +void attention_forward(float* out, float* qkvr, float* att, float* inp, int B, int T, int C, int NH) { + int HS = C / NH; + + float *q, *k, *v; + q = qkvr + 0 * B * T * C; + k = qkvr + 1 * B * T * C; + v = qkvr + 2 * B * T * C; + + dim3 block(256, 1, 1); + dim3 grid((B * NH * T * HS + block.x - 1) / block.x, 1, 1); + permute_kernel<<>>(q, k, v, inp, B, T, NH, HS); + + // 1. Check if KV-caching is enabled, used by the verification script + if (!g_enable_kv_cache) { + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x - 1) / block.x, 1, 1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0f / sqrtf((float)HS); + dim3 grid_softmax((B * NH * T + block.x - 1) / block.x, 1, 1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + dim3 grid_attv((B * NH * T * HS + block.x - 1) / block.x); // <<< ADD + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); // <<< grid_attv not grid_preatt + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); + return; + } + + allocate_kv_cache(B, NH, HS); // Ensure KV cache is allocated with the right dimensions + // store K and V in the cache + store_kv_kernel<<>>(g_kv_cache, k, v, g_layer_idx, B, NH, HS, T, g_current_pos, g_is_prefill); + + + if (g_is_prefill) { + // --- PREFILL PHASE --- + // During prefill, we process the entire input sequence (T > 1). + + // A. Store the K and V results from this layer into the cache. + // The projections are available in 'inp' (the QKV matmul output). + + // B. Run standard baseline attention for the initial prompt tokens. + + // C. Update the global layer index to track progress through the model. + + float* preatt = inp; + dim3 grid_preatt((B * NH * T * T + block.x -1) / block.x, 1, 1); + preattention_scores_kernel<<>>(q, k, B, T, NH, HS, preatt); + + float scale = 1.0f / sqrtf((float)HS); + dim3 grid_softmax((B * NH * T + block.x -1) / block.x, 1, 1); + softmax_forward_kernel<<>>(att, scale, preatt, B * NH, T); + + float* vaccum = inp; + dim3 grid_attv((B * NH * T * HS + block.x - 1) / block.x); + attention_v_kernel<<>>(vaccum, att, v, B, T, NH, HS); + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); + + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_is_prefill = false; // Transition to Decode phase for the next token + g_current_pos = T; // Set current position to the end of the prefill sequence + } + } else { + // --- DECODE PHASE --- + // During decode, we only process the single newest token (T = 1). + + // A. Store the single new Key and Value for this step into the cache at g_current_pos. + + // B. Run specialized Decode Attention. + // This requires reading the history of Keys and Values from the cache + // to compute the output for the current token. + // TODO: Don't forget to scale your attention scores + // TODO: Ensure the output shapes match the next layer + + // C. Update the global state. + + float *vaccum = inp; + dim3 grid_dec((B * NH * HS + block.x - 1) / block.x); + decode_attention_kernel<<>>(vaccum, q, g_kv_cache, g_layer_idx, B, NH, HS, g_current_pos); + unpermute_kernel<<>>(vaccum, out, B, T, NH, HS); + + g_layer_idx++; + if (g_layer_idx == 12) { + g_layer_idx = 0; + g_current_pos++; // Advance the sequence position + } + } +} + +#endif // __ATTENTION_KV_CACHE_TEMPLATE_CUH__ + diff --git a/local_attn_verify.cu b/local_attn_verify.cu new file mode 100644 index 0000000..8c4257a --- /dev/null +++ b/local_attn_verify.cu @@ -0,0 +1,215 @@ +#include +#include +#include +#include +#include + +#include "cpu_kernels/local_attention.cuh" + +//! TODO: Include your local attention kernel here! +#include "kernels/local_attention.cuh" + +#define cudaCheck(call) do { \ + cudaError_t err = call; \ + if (err != cudaSuccess) { \ + fprintf(stderr, "CUDA error in %s at line %d: %s\n", __FILE__, __LINE__, cudaGetErrorString(err)); \ + exit(EXIT_FAILURE); \ + } \ +} while(0) + +void read_input_from_file(const char* filename, int* batch_size, int* seq_length, int* num_heads, int* head_dim, float** input) { + FILE* file = fopen(filename, "r"); + if (!file) { + fprintf(stderr, "Error opening input file: %s\n", filename); + exit(EXIT_FAILURE); + } + + fscanf(file, "%d\n%d\n%d\n%d", batch_size, seq_length, num_heads, head_dim); + + int B = *batch_size; int T = *seq_length; + int NH = *num_heads; int HS = *head_dim; + *input = (float*) malloc(B * T * 3 * NH * HS * sizeof(float)); + + // read Q, K, V + for (int b = 0; b < B; ++b) { + for (int t = 0; t < T; ++t) { + for (int nh = 0; nh < NH; ++nh) { + for (int hs = 0; hs < HS; ++hs) { + int idx = b * T * 3 * NH * HS + t * 3 * NH * HS + 0 * NH * HS + nh * HS + hs; + fscanf(file, "%f", &((*input)[idx])); + } + } + } + } + fgetc(file); + for (int b = 0; b < B; ++b) { + for (int t = 0; t < T; ++t) { + for (int nh = 0; nh < NH; ++nh) { + for (int hs = 0; hs < HS; ++hs) { + int idx = b * T * 3 * NH * HS + t * 3 * NH * HS + 1 * NH * HS + nh * HS + hs; + fscanf(file, "%f", &((*input)[idx])); + } + } + } + } + fgetc(file); + for (int b = 0; b < B; ++b) { + for (int t = 0; t < T; ++t) { + for (int nh = 0; nh < NH; ++nh) { + for (int hs = 0; hs < HS; ++hs) { + int idx = b * T * 3 * NH * HS + t * 3 * NH * HS + 2 * NH * HS + nh * HS + hs; + fscanf(file, "%f", &((*input)[idx])); + } + } + } + } + fclose(file); +} + +void read_output_from_file(const char* filename, int* batch_size, int* seq_length, int* num_heads, int* head_dim, float** outputs) { + FILE* file = fopen(filename, "r"); + if (!file) { + fprintf(stderr, "Error opening input file: %s\n", filename); + exit(EXIT_FAILURE); + } + + fscanf(file, "%d\n%d\n%d\n%d", batch_size, seq_length, num_heads, head_dim); + + int tensor_size = (*batch_size) * (*seq_length) * (*num_heads) * (*head_dim); + *outputs = (float*) malloc(tensor_size * sizeof(float)); + + for (int i = 0; i < tensor_size; ++i) fscanf(file, "%f", &((*outputs)[i])); + fgetc(file); + fclose(file); +} + +void compare_arrays(float* computed_outputs, float* expected_outputs, int size, const char* test_name) { + int diff_count = 0; + double max_relative_error = 0.0; + double rmse = 0.0; + int first_mismatch_idx = -1; + const double ERROR_THRESHOLD = 0.1; // 10% relative error threshold + + for (int i = 0; i < size; ++i) { + double abs_error = fabs(computed_outputs[i] - expected_outputs[i]); + double rel_error = abs_error / (fabs(expected_outputs[i]) + 1e-6); + rmse += abs_error * abs_error; + + if (rel_error > max_relative_error) { + max_relative_error = rel_error; + } + + if (rel_error > ERROR_THRESHOLD) { + if (first_mismatch_idx == -1) { + first_mismatch_idx = i; + } + ++diff_count; + } + } + + rmse = sqrt(rmse / size); + + if (diff_count > 0) { + printf("+++++++++++++++ %s TEST FAILED with %d differing values (%.2f%%) +++++++++++++++\n", + test_name, diff_count, (100.0 * diff_count) / size); + printf("Max Relative Error: %.6f, RMSE: %.6f\n", max_relative_error, rmse); + + if (first_mismatch_idx >= 0) { + printf("First mismatch at index %d: Expected %.6f, Got %.6f\n", + first_mismatch_idx, expected_outputs[first_mismatch_idx], computed_outputs[first_mismatch_idx]); + } + } else { + printf("-------------------- Test Passed! (Max Relative Error: %.6f) --------------------\n", + max_relative_error); + } +} + +int main() { + printf("============= Local Attention Skeleton Test Code (Inputs and Golden Solution Provided) =============\n"); + + cudaCheck(cudaSetDevice(0)); + + int num_examples = 8; + + for (int example_idx = 0; example_idx < num_examples; example_idx++) { + int batch_size = 0; + int seq_length = 0; + int num_heads = 0; + int head_dim = 0; + + float *input_data = NULL; + float *expected_outputs = NULL; + + char input_filepath[512]; + char output_filepath[512]; + snprintf(input_filepath, sizeof(input_filepath), "/work/hdd/bche/Project_GPT/local_attn_examples/example_%i.tcin", example_idx); + snprintf(output_filepath, sizeof(output_filepath), "/work/hdd/bche/Project_GPT/local_attn_examples/example_%i.tcout", example_idx); + + printf("\n Testing Example %d: \n", example_idx); + + // read provided input + read_input_from_file(input_filepath, &batch_size, &seq_length, &num_heads, &head_dim, &input_data); + // read golden solution values + read_output_from_file(output_filepath, &batch_size, &seq_length, &num_heads, &head_dim, &expected_outputs); + + // --------------- CPU reference implementation test example --------------- + + int size = batch_size * seq_length * num_heads * head_dim; + + // separate copy of input for CPU to avoid input memory being overwritten + float* att_inp_cpu = (float*) malloc(size * 3 * sizeof(float)); + float* qkvr_local = (float*) malloc(size * 3 * sizeof(float)); + float* att_out = (float*) malloc(size * sizeof(float)); + + memcpy(att_inp_cpu, input_data, size * 3 * sizeof(float)); + + // calls the CPU implementation to compute windowed attention outputs + local_attention_forward_cpu(att_out, qkvr_local, att_inp_cpu, batch_size, seq_length, num_heads, head_dim); + + //! TODO: uncomment the below line to see how to compare outputs of the provided CPU code with golden solution + // compare_arrays(att_out, expected_outputs, size, input_filepath); + + free(att_inp_cpu); + free(qkvr_local); + free(att_out); + + //! TODO: Modify the below code to test your GPU implementation! + + // --------------- Your GPU implementation test --------------- + + int tensor_size = batch_size * seq_length * num_heads * head_dim; + + float *device_input = NULL; + float *device_qkvr = NULL; + float *device_output = NULL; + + cudaCheck(cudaMalloc(&device_input, tensor_size * 3 * sizeof(float))); + cudaCheck(cudaMalloc(&device_qkvr, tensor_size * 3 * sizeof(float))); + cudaCheck(cudaMalloc(&device_output, tensor_size * sizeof(float))); + cudaCheck(cudaMemcpy(device_input, input_data, tensor_size * 3 * sizeof(float), + cudaMemcpyHostToDevice)); + // //! TODO: Call your GPU implementation here (including permute and unpermute) + local_attention_forward_gpu(device_output, device_qkvr, NULL, device_input, batch_size, seq_length, num_heads, head_dim); + + cudaCheck(cudaGetLastError()); + cudaCheck(cudaDeviceSynchronize()); + + // // Copy results back to host for comparison + float* host_outputs = (float*)calloc(tensor_size, sizeof(float)); + cudaCheck(cudaMemcpy(host_outputs, device_output, tensor_size * sizeof(float), + cudaMemcpyDeviceToHost)); + + compare_arrays(host_outputs, expected_outputs, tensor_size, input_filepath); + + // //! TODO: Free allocated memory in host and device + free(input_data); + free(expected_outputs); + free(host_outputs); + cudaCheck(cudaFree(device_input)); + cudaCheck(cudaFree(device_qkvr)); + cudaCheck(cudaFree(device_output)); + } + + printf("\n================== End of Tests Reached ==================\n"); + return 0; +} \ No newline at end of file diff --git a/local_attn_verify.slurm b/local_attn_verify.slurm new file mode 100644 index 0000000..165d45c --- /dev/null +++ b/local_attn_verify.slurm @@ -0,0 +1,20 @@ +#!/bin/bash +#SBATCH --job-name="local_attn_verify" +#SBATCH --output="local_attn_verify.out" +#SBATCH --error="local_attn_verify.err" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:25:00 + +module reset + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +srun ./local_attn_verify \ No newline at end of file diff --git a/next_token_generation.cu b/next_token_generation.cu new file mode 100644 index 0000000..c805f48 --- /dev/null +++ b/next_token_generation.cu @@ -0,0 +1,293 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// GPU / CUDA related +#include +#include +#include "utils/utils.h" +#include "utils/tokenizer.h" +#include "utils/cuda_utils.cuh" + +#include "gpt2.cuh" + +const char *generation_output_filename = "generation_outputs.txt"; +const char *checkpoint_file = "/work/hdd/bche/Project_GPT/gpt2_124M.bin"; +#define SEQUENCE_GENERATION_BATCH_SIZE 1 /* 1 for sequence generation */ +#define TEMPERATURE 0.7 /* Should be > 0 */ + +/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ HELPER FUNCTIONS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */ + +void get_substring_up_to_space(char *input, char *output) { + char *space_pos = strchr(input + 1, ' '); + + if (space_pos != NULL) { + size_t length = space_pos - input; + strncpy(output, input, length); + // null-terminate output + output[length] = '\0'; + } else { + // if no space, copy the entire string + strcpy(output, input); + } +} + +int* tokenizer_encode(Tokenizer *tokenizer, const char *input_str, int *num_tokens) { + if (tokenizer->init_ok == 0) { + *num_tokens = 0; + return NULL; + } + + // arbitrary initial number of tokens + int max_tokens = 128; + int *tokens = (int*)mallocCheck(max_tokens * sizeof(int)); + int token_count = 0; + + char temp_str[strlen(input_str) + 1]; + strcpy(temp_str, input_str); + + // pointer to keep track of the remaining string to process + char *remaining_input_str = temp_str; + while (*remaining_input_str != '\0') { + char to_search[100]; + get_substring_up_to_space(remaining_input_str, to_search); + size_t token_len = strlen(to_search); + + // try to find longest matching token from the token_table (not perfect, but probably good enough heuristic) + int best_token = -1; + int best_token_length = -1; + for (int i = 0; i < tokenizer->vocab_size; ++i) { + const char *token = tokenizer->token_table[i]; + int tokenlen = strlen(token); + + if (strncmp(to_search, token, tokenlen) == 0) { + if (tokenlen > best_token_length) { + best_token_length = tokenlen; + best_token = i; + } + } + } + assert(best_token != 0); + tokens[token_count++] = best_token; + // move the pointer ahead by the length of the token + remaining_input_str += best_token_length; + + // if reached current tokens array capacity, resize + if (token_count >= max_tokens) { + // double the capacity + max_tokens *= 2; + tokens = (int*)realloc(tokens, max_tokens * sizeof(int)); + if (!tokens) { + fprintf(stderr, "next_token_gen - Memory allocation failed during token array resizing.\n"); + exit(EXIT_FAILURE); + } + } + } + + *num_tokens = token_count; + return tokens; +} + +int return_random_token(int vocab_size) { + int lower_bound = 0; + int upper_bound = vocab_size - 1; + return rand() % (upper_bound - lower_bound + 1) + lower_bound; +} + +int sample_next_token_temperature(const float* logits, int vocab_size, float temperature) { + // find max logit, to be used for numerical stability + float max_logit = -FLT_MAX; + for (int i = 0; i < vocab_size; i++) { + if (logits[i] > max_logit) { + max_logit = logits[i]; + } + } + + float norm = 0.0f; + float* probabilities = (float*)mallocCheck(vocab_size * sizeof(float)); + + // compute probabilities using softmax with numerical stability and temperature + for (int i = 0; i < vocab_size; i++) { + probabilities[i] = expf((logits[i] - max_logit) / temperature); + norm += probabilities[i]; + } + + // normalize + for (int i = 0; i < vocab_size; i++) { + probabilities[i] /= norm; + } + + // sample from distribution + float random_value = (float)rand() / RAND_MAX; + float cumulative = 0.0f; + + for (int i = 0; i < vocab_size; ++i) { + cumulative += probabilities[i]; + if (random_value < cumulative) { + free(probabilities); + return i; + } + } + + free(probabilities); + + // default fallback + return vocab_size - 1; // return last token in vocab + // return return_random_token(vocab_size); // return a random token in vocab +} + +void append_to_file(const char *filename, const char *string_to_append) { + FILE *file = fopenCheck(filename, "a"); + if (file == NULL) { + perror("next_token_gen - Error opening file"); + exit(EXIT_FAILURE); + } + fprintf(file, "%s", string_to_append); + fclose(file); +} + +void log_generation_output(const char* input_sequence, const char* output_text) { + append_to_file(generation_output_filename, "next_token_gen - INPUT TEXT:\n"); + append_to_file(generation_output_filename, input_sequence); + append_to_file(generation_output_filename, "\n\nGENERATED TEXT:\n"); + append_to_file(generation_output_filename, output_text); + append_to_file(generation_output_filename, "\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n"); +} + +/* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */ + +void gpt2_generate_next_token(GPT2* model, int* input_tokens, int input_length, int* output_token) { + assert(input_length <= model->config.max_seq_len); + + int padded_input_length = (input_length + 3) / 4 * 4; + int * gpt_input = (int *)mallocCheck(padded_input_length * sizeof(int)); + memcpy(gpt_input, input_tokens, input_length * sizeof(int)); + if (padded_input_length - input_length > 0) + memset(gpt_input + input_length, 0, (padded_input_length - input_length) * sizeof(int)); + gpt2_forward(model, gpt_input, SEQUENCE_GENERATION_BATCH_SIZE, padded_input_length); + + float* logits_gpu = (model->acts.output) + (input_length - 1) * model->config.padded_vocab_size; + float* logits_cpu = (float*)mallocCheck(model->config.vocab_size * sizeof(float)); + // move logits back to CPU and sample (note we only move the first vocab_size logits, ignoring the padding) + cudaCheck(cudaMemcpy(logits_cpu, logits_gpu, model->config.vocab_size * sizeof(float), cudaMemcpyDeviceToHost)); + + *output_token = sample_next_token_temperature(logits_cpu, model->config.vocab_size, TEMPERATURE); + + free(logits_cpu); +} + +void gpt2_generate_text(GPT2* model, const char* input_sequence, int max_gen_length, char* output_text) { + Tokenizer tokenizer; + tokenizer_init(&tokenizer, "/work/hdd/bche/Project_GPT/gpt2_tokenizer.bin"); + + int num_input_tokens = 0; + int* input_tokens = tokenizer_encode(&tokenizer, input_sequence, &num_input_tokens); + printf("next_token_gen - number of input tokens: %d\n", num_input_tokens); + for (int i = 0; i < num_input_tokens; i++) { + printf("%d ", input_tokens[i]); + } + + // the model's response is actually at generated_sequence + num_input_tokens + int* generated_sequence = (int*)mallocCheck((num_input_tokens + max_gen_length) * sizeof(int)); + memcpy(generated_sequence, input_tokens, num_input_tokens * sizeof(int)); + + // seq_len_itr accounts for the number of input tokens + // i.e. put new tokens into this index due to 0-indexing + int seq_len_itr = num_input_tokens; + + // if you want to include prompt in generated output text string + // strcpy(output_text, input_sequence); + + for (int i = 0; i < max_gen_length; ++i) { + int next_token; + gpt2_generate_next_token(model, generated_sequence, seq_len_itr, &next_token); + + if (seq_len_itr < model->config.max_seq_len) { + generated_sequence[seq_len_itr] = next_token; + seq_len_itr += 1; + } + + } + + // this should not be segfaulting if max_gen_length number of tokens were generated properly + for (int i = num_input_tokens; i < num_input_tokens + max_gen_length; ++i) { + // decode the token and append to output text + const char* next_token_str = tokenizer_decode(&tokenizer, generated_sequence[i]); + strcat(output_text, next_token_str); + } + + free(input_tokens); + free(generated_sequence); + tokenizer_free(&tokenizer); +} + +int main(int argc, char** argv) { + if (argc < 2) { + printf("next_token_gen - Usage: %s \"\"\n", argv[0]); + return EXIT_FAILURE; + } + + srand(time(NULL)); + + // setup cuBLAS and cuBLASLt + cublasCheck(cublasCreate(&cublas_handle)); + // TF32 precision is equivalent to torch.set_float32_matmul_precision('high') + int enable_tf32 = 0; + cublas_compute_type = enable_tf32 ? CUBLAS_COMPUTE_32F_FAST_TF32 : CUBLAS_COMPUTE_32F; + cublasMath_t cublas_math_mode = enable_tf32 ? CUBLAS_TF32_TENSOR_OP_MATH : CUBLAS_DEFAULT_MATH; + cublasCheck(cublasSetMathMode(cublas_handle, cublas_math_mode)); + + GPT2 model; + gpt2_build_from_checkpoint(&model, checkpoint_file); + printf("+-----------------------+\n"); + printf("| GPT2 MODEL PARAMETERS |\n"); + printf("+-----------------------+----------------------------------------------------+\n"); + printf("| max_sequence_length T | %-50d |\n", model.config.max_seq_len); + printf("| vocab_size V | %-50d |\n", model.config.vocab_size); + printf("| padded_vocab_size Vp | %-50d |\n", model.config.padded_vocab_size); + printf("| num_layers L | %-50d |\n", model.config.num_layers); + printf("| num_heads NH | %-50d |\n", model.config.num_heads); + printf("| channels C | %-50d |\n", model.config.channels); + printf("| num_parameters | %-50zu |\n", model.num_parameters); + printf("+-----------------------+----------------------------------------------------+\n"); + + // max_gen_length DOES NOT account for the input sequence length + // 500 runs out of memory + int max_gen_length = 50; + + const char* input_sequence = argv[1]; + + char output_text[4096] = {0}; + + struct timespec start, end; + clock_gettime(CLOCK_MONOTONIC, &start); + gpt2_generate_text(&model, input_sequence, max_gen_length, output_text); + clock_gettime(CLOCK_MONOTONIC, &end); + double time_elapsed_s = (end.tv_sec - start.tv_sec) + (end.tv_nsec - start.tv_nsec) / 1e9; + + printf("+----------+\n"); + printf("| GPT2 RUN |\n"); + printf("+----------+----------------------------------------------------------------------------------------------------------------------------------+\n"); + printf("|\n| INPUT TEXT: %s\n", input_sequence); + printf("|\n| GENERATED TEXT: %s\n", output_text); + printf("|\n| TIME FOR INFERENCING: %f ms\n", time_elapsed_s * 1000); + printf("|\n| # TOKENS/SEC: %.3f\n", max_gen_length / time_elapsed_s); + printf("+---------------------------------------------------------------------------------------------------------------------------------------------+\n"); + + log_generation_output(input_sequence, output_text); + + return EXIT_SUCCESS; +} diff --git a/old_verify.slurm b/old_verify.slurm new file mode 100644 index 0000000..b70f4bd --- /dev/null +++ b/old_verify.slurm @@ -0,0 +1,38 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --output="verify_output.out" +#SBATCH --error="verify_errors.err" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:25:00 + +module reset + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +# run random kernel output verification +# srun ./output_verification_rand + +# run kernel output verification against gold reference +# srun ./test_gpt2_kernels + +# run overall inference test +srun ./test_gpt2 + +# =========================== Profiling =========================== +# To use profiling tools, uncomment the below commands, and make sure +# your constraint parameter is set to "projects,perf,nvperf" +# Please make sure to set it back to "projects" for standard testing. + +# --Generate nsys profile information, output will be in test_gpt2_nsys_prof.out-- +# srun nsys profile --stats=true ./test_gpt2 > test_gpt2_nsys_prof.out + +# --Generate analysis_file.ncu-rep with a detailed profiling result--- +# ncu --set full -f -o analysis_file ./test_gpt2 > test_gpt2.out \ No newline at end of file diff --git a/old_verify_new.slurm b/old_verify_new.slurm new file mode 100644 index 0000000..f9f7102 --- /dev/null +++ b/old_verify_new.slurm @@ -0,0 +1,28 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --output="verify_output.out" +#SBATCH --error="verify_errors.err" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects,perf,nvperf" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:15:00 +#SBATCH --exclude=gpub029,gpub034,gpub038,gpub043,gpub074,gpub054,gpub012 +#SBATCH --mail-user=asing271@illinois.edu +#SBATCH --mail-type=BEGIN,END,FAIL + +module reset +module load cuda + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +# ── nsys system-level profile ───────────────────────────────────── +srun nsys profile --stats=true ./test_gpt2 > test_gpt2_nsys_prof_req_0.out + +# ── ncu kernel-level profile ────────────────────────────────────── +# ncu --set full -f -o analysis_file ./test_gpt2 > test_gpt2_ncu.out \ No newline at end of file diff --git a/output/perf/ncu/verify_op_11.err b/output/perf/ncu/verify_op_11.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_11.out b/output/perf/ncu/verify_op_11.out new file mode 100644 index 0000000..10bc526 --- /dev/null +++ b/output/perf/ncu/verify_op_11.out @@ -0,0 +1,173 @@ +job 18002677 starting on gpub089.delta.ncsa.illinois.edu + + +==PROF== Connected to process 810795 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_11) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 146: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 810795 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_11.ncu-rep diff --git a/output/perf/ncu/verify_op_12.err b/output/perf/ncu/verify_op_12.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_12.out b/output/perf/ncu/verify_op_12.out new file mode 100644 index 0000000..e175001 --- /dev/null +++ b/output/perf/ncu/verify_op_12.out @@ -0,0 +1,197 @@ +job 18002228 starting on gpub022.delta.ncsa.illinois.edu + + +==PROF== Connected to process 355678 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_12) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 146: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 147: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 148: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 149: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 150: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 151: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 152: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 153: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 154: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 155: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 156: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 162: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 163: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 164: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 165: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 166: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 167: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 168: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 169: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_kernel" - 170: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431793 +-39.836346, -39.836502 +-43.065910, -43.066078 +-42.828045, -42.828197 +-43.529541, -43.529701 +-44.318398, -44.318558 +-41.227425, -41.227592 +-41.270760, -41.270931 +-42.541393, -42.541584 +-42.394997, -42.395176 +---------------- Logits OK! (max err 0.000050, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 355678 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_12.ncu-rep diff --git a/output/perf/ncu/verify_op_13.err b/output/perf/ncu/verify_op_13.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_13.out b/output/perf/ncu/verify_op_13.out new file mode 100644 index 0000000..be915b5 --- /dev/null +++ b/output/perf/ncu/verify_op_13.out @@ -0,0 +1,197 @@ +job 18003871 starting on gpub081.delta.ncsa.illinois.edu + + +==PROF== Connected to process 459778 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_13) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 146: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 147: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 148: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 149: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 150: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 151: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 152: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 153: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 154: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 155: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 156: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 162: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 163: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 164: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 165: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 166: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 167: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 168: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 169: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 170: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 459778 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_13.ncu-rep diff --git a/output/perf/ncu/verify_op_16.err b/output/perf/ncu/verify_op_16.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_16.out b/output/perf/ncu/verify_op_16.out new file mode 100644 index 0000000..b8745d9 --- /dev/null +++ b/output/perf/ncu/verify_op_16.out @@ -0,0 +1,231 @@ +job 18004587 starting on gpub005.delta.ncsa.illinois.edu + + +==PROF== Connected to process 408869 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_16) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 146: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 147: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 148: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 149: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 150: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 151: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 152: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 153: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 154: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 155: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 156: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 162: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 163: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 164: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 165: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 166: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 167: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 168: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 169: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 170: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 171: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 172: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 173: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 174: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 175: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 176: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 177: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 178: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 179: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 180: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 181: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 182: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 183: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 184: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 185: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 186: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 187: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 188: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 189: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 190: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 191: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 192: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 193: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 194: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 195: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 196: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 197: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 198: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 199: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 200: 0%....50%....100% - 40 passes +==PROF== Profiling "Kernel" - 201: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_64x64_tn" - 202: 0%....50%....100% - 40 passes +==PROF== Profiling "bias_add_filler_kernel" - 203: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 204: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 205: 0%....50%....100% - 40 passes +==PROF== Profiling "ampere_sgemm_128x64_tn" - 206: 0%....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.315117 +-39.836346, -39.732384 +-43.065910, -42.960098 +-42.828045, -42.723515 +-43.529541, -43.418613 +-44.318398, -44.210033 +-41.227425, -41.119377 +-41.270760, -41.164490 +-42.541393, -42.428894 +-42.394997, -42.288929 +---------------- Logits OK! (max err 0.082334, RMSE 0.002728) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 408869 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_16.ncu-rep diff --git a/output/perf/ncu/verify_op_8.err b/output/perf/ncu/verify_op_8.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_8.out b/output/perf/ncu/verify_op_8.out new file mode 100644 index 0000000..2a0f41e --- /dev/null +++ b/output/perf/ncu/verify_op_8.out @@ -0,0 +1,197 @@ +job 17968511 starting on gpub005.delta.ncsa.illinois.edu + + +==PROF== Connected to process 3198281 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_8) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 146: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 147: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 148: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 149: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 150: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 151: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 152: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 153: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 154: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 155: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 156: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 162: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 163: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 164: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 165: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 166: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 167: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 168: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 169: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 170: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 3198281 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_8.ncu-rep diff --git a/output/perf/ncu/verify_op_9.err b/output/perf/ncu/verify_op_9.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_op_9.out b/output/perf/ncu/verify_op_9.out new file mode 100644 index 0000000..9dd3841 --- /dev/null +++ b/output/perf/ncu/verify_op_9.out @@ -0,0 +1,197 @@ +job 17968384 starting on gpub015.delta.ncsa.illinois.edu + + +==PROF== Connected to process 1935467 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_9) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 146: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 147: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 148: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 149: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 150: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 151: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 152: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 153: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 154: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 155: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 156: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes +==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes +==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes +==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 162: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 163: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 164: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 165: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 166: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 167: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 168: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 169: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 170: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 1935467 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_9.ncu-rep diff --git a/output/perf/ncu/verify_req_4.err b/output/perf/ncu/verify_req_4.err new file mode 100644 index 0000000..e69de29 diff --git a/output/perf/ncu/verify_req_4.out b/output/perf/ncu/verify_req_4.out new file mode 100644 index 0000000..2450027 --- /dev/null +++ b/output/perf/ncu/verify_req_4.out @@ -0,0 +1,173 @@ +job 17971939 starting on gpub022.delta.ncsa.illinois.edu + + +==PROF== Connected to process 3614659 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_req_4) +==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 2: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 3: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 4: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 5: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 6: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 7: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 8: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 9: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 10: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 11: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 12: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 13: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 14: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 15: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 16: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 17: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 18: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 19: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 20: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 21: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 22: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 23: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 24: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 25: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 26: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 27: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 28: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 29: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 30: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 31: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 32: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 33: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 34: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 35: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 36: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 37: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 38: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 39: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 40: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 41: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 42: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 43: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 44: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 45: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 46: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 47: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 48: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 49: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 50: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 51: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 52: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 53: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 54: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 55: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 56: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 57: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 58: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 59: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 60: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 61: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 62: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 63: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 64: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 65: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 66: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 67: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 68: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 69: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 70: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 71: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 72: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 73: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 74: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 75: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 76: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 77: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 78: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 81: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 82: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 83: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 84: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 85: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 86: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 87: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 88: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 89: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 90: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 91: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 92: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 93: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 94: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 95: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 96: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 97: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 98: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 99: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 100: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 101: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 102: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 103: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 104: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 105: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 106: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 107: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 108: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 109: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 110: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 111: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 112: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 113: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 114: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 115: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 116: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 117: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 118: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 119: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 120: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 121: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 122: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 123: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 124: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 125: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 126: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 127: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 128: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 129: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 130: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 131: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 132: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 133: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 134: 0%....50%....100% - 40 passes +==PROF== Profiling "permute_kernel" - 135: 0%....50%....100% - 40 passes +==PROF== Profiling "flash_attention_forward_kernel" - 136: 0%....50%....100% - 40 passes +==PROF== Profiling "unpermute_kernel" - 137: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 138: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 139: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 140: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 141: 0%....50%....100% - 40 passes +==PROF== Profiling "gelu_forward_kernel" - 142: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 143: 0%....50%....100% - 40 passes +==PROF== Profiling "residual_forward_kernel" - 144: 0%....50%....100% - 40 passes +==PROF== Profiling "layernorm_forward_kernel" - 145: 0%....50%....100% - 40 passes +==PROF== Profiling "matmul_forward_kernel" - 146: 0% +==WARNING== Launching the workload is taking more time than expected. If this continues to hang, terminate the profile and re-try by profiling the range of all related launches using '--replay-mode range'. See https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#replay for more details. +....50%....100% - 40 passes +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +==PROF== Disconnected from process 3614659 +==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_req_4.ncu-rep diff --git a/output/perf/nsys/verify_op_11.err b/output/perf/nsys/verify_op_11.err new file mode 100644 index 0000000..f82cc28 --- /dev/null +++ b/output/perf/nsys/verify_op_11.err @@ -0,0 +1,9 @@ +WARNING: Device-side CUDA Event completion trace is currently enabled. + This may increase runtime overhead and the likelihood of false + dependencies across CUDA Streams. If you wish to avoid this, please + disable the feature with --cuda-event-trace=false. +Failed to create '/u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_11.nsys-rep': File exists. +Use `--force-overwrite true` to overwrite existing files. +Failed to create '/u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_11.sqlite': File exists. +Use `--force-overwrite true` to overwrite existing files. +SKIPPED: /tmp/nsys-report-472f.sqlite does not contain NV Tools Extension (NVTX) data. diff --git a/output/perf/nsys/verify_op_11.out b/output/perf/nsys/verify_op_11.out new file mode 100644 index 0000000..6e87078 --- /dev/null +++ b/output/perf/nsys/verify_op_11.out @@ -0,0 +1,107 @@ +job 18002676 starting on gpub022.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-4b2b.qdstrm' + [1/8] [0% ] nsys-report-d663.nsys-rep [1/8] [0% ] nsys-report-d663.nsys-rep [1/8] [6% ] nsys-report-d663.nsys-rep [1/8] [=====29% ] nsys-report-d663.nsys-rep [1/8] [===========52% ] nsys-report-d663.nsys-rep [1/8] [===========53% ] nsys-report-d663.nsys-rep [1/8] [=====32% ] nsys-report-d663.nsys-rep [1/8] [===========52% ] nsys-report-d663.nsys-rep [1/8] [===========53% ] nsys-report-d663.nsys-rep [1/8] [============54% ] nsys-report-d663.nsys-rep [1/8] [============57% ] nsys-report-d663.nsys-rep [1/8] [=============58% ] nsys-report-d663.nsys-rep [1/8] [=============59% ] nsys-report-d663.nsys-rep [1/8] [=================74% ] nsys-report-d663.nsys-rep [1/8] [======================92% ] nsys-report-d663.nsys-rep [1/8] [========================100%] nsys-report-d663.nsys-rep [1/8] [========================100%] nsys-report-d663.nsys-rep + [2/8] [0% ] nsys-report-472f.sqlite [2/8] [1% ] nsys-report-472f.sqlite [2/8] [2% ] nsys-report-472f.sqlite [2/8] [3% ] nsys-report-472f.sqlite [2/8] [4% ] nsys-report-472f.sqlite [2/8] [5% ] nsys-report-472f.sqlite [2/8] [6% ] nsys-report-472f.sqlite [2/8] [7% ] nsys-report-472f.sqlite [2/8] [8% ] nsys-report-472f.sqlite [2/8] [9% ] nsys-report-472f.sqlite [2/8] [10% ] nsys-report-472f.sqlite [2/8] [11% ] nsys-report-472f.sqlite [2/8] [12% ] nsys-report-472f.sqlite [2/8] [13% ] nsys-report-472f.sqlite [2/8] [14% ] nsys-report-472f.sqlite [2/8] [=15% ] nsys-report-472f.sqlite [2/8] [=16% ] nsys-report-472f.sqlite [2/8] [=17% ] nsys-report-472f.sqlite [2/8] [==18% ] nsys-report-472f.sqlite [2/8] [==19% ] nsys-report-472f.sqlite [2/8] [==20% ] nsys-report-472f.sqlite [2/8] [==21% ] nsys-report-472f.sqlite [2/8] [===22% ] nsys-report-472f.sqlite [2/8] [===23% ] nsys-report-472f.sqlite [2/8] [===24% ] nsys-report-472f.sqlite [2/8] [====25% ] nsys-report-472f.sqlite [2/8] [====26% ] nsys-report-472f.sqlite [2/8] [====27% ] nsys-report-472f.sqlite [2/8] [====28% ] nsys-report-472f.sqlite [2/8] [=====29% ] nsys-report-472f.sqlite [2/8] [=====30% ] nsys-report-472f.sqlite [2/8] [=====31% ] nsys-report-472f.sqlite [2/8] [=====32% ] nsys-report-472f.sqlite [2/8] [======33% ] nsys-report-472f.sqlite [2/8] [======34% ] nsys-report-472f.sqlite [2/8] [======35% ] nsys-report-472f.sqlite [2/8] [=======36% ] nsys-report-472f.sqlite [2/8] [=======37% ] nsys-report-472f.sqlite [2/8] [=======38% ] nsys-report-472f.sqlite [2/8] [=======39% ] nsys-report-472f.sqlite [2/8] [========40% ] nsys-report-472f.sqlite [2/8] [========41% ] nsys-report-472f.sqlite [2/8] [========42% ] nsys-report-472f.sqlite [2/8] [=========43% ] nsys-report-472f.sqlite [2/8] [=========44% ] nsys-report-472f.sqlite [2/8] [=========45% ] nsys-report-472f.sqlite [2/8] [=========46% ] nsys-report-472f.sqlite [2/8] [==========47% ] nsys-report-472f.sqlite [2/8] [==========48% ] nsys-report-472f.sqlite [2/8] [==========49% ] nsys-report-472f.sqlite [2/8] [===========50% ] nsys-report-472f.sqlite [2/8] [===========51% ] nsys-report-472f.sqlite [2/8] [===========52% ] nsys-report-472f.sqlite [2/8] [===========53% ] nsys-report-472f.sqlite [2/8] [============54% ] nsys-report-472f.sqlite [2/8] [============55% ] nsys-report-472f.sqlite [2/8] [============56% ] nsys-report-472f.sqlite [2/8] [============57% ] nsys-report-472f.sqlite [2/8] [=============58% ] nsys-report-472f.sqlite [2/8] [=============59% ] nsys-report-472f.sqlite [2/8] [=============60% ] nsys-report-472f.sqlite [2/8] [==============61% ] nsys-report-472f.sqlite [2/8] [==============62% ] nsys-report-472f.sqlite [2/8] [==============63% ] nsys-report-472f.sqlite [2/8] [==============64% ] nsys-report-472f.sqlite [2/8] [===============65% ] nsys-report-472f.sqlite [2/8] [===============66% ] nsys-report-472f.sqlite [2/8] [===============67% ] nsys-report-472f.sqlite [2/8] [================68% ] nsys-report-472f.sqlite [2/8] [================69% ] nsys-report-472f.sqlite [2/8] [================70% ] nsys-report-472f.sqlite [2/8] [================71% ] nsys-report-472f.sqlite [2/8] [=================72% ] nsys-report-472f.sqlite [2/8] [=================73% ] nsys-report-472f.sqlite [2/8] [=================74% ] nsys-report-472f.sqlite [2/8] [==================75% ] nsys-report-472f.sqlite [2/8] [==================76% ] nsys-report-472f.sqlite [2/8] [==================77% ] nsys-report-472f.sqlite [2/8] [==================78% ] nsys-report-472f.sqlite [2/8] [===================79% ] nsys-report-472f.sqlite [2/8] [===================80% ] nsys-report-472f.sqlite [2/8] [===================81% ] nsys-report-472f.sqlite [2/8] [===================82% ] nsys-report-472f.sqlite [2/8] [====================83% ] nsys-report-472f.sqlite [2/8] [====================84% ] nsys-report-472f.sqlite [2/8] [====================85% ] nsys-report-472f.sqlite [2/8] [=====================86% ] nsys-report-472f.sqlite [2/8] [=====================87% ] nsys-report-472f.sqlite [2/8] [=====================88% ] nsys-report-472f.sqlite [2/8] [=====================89% ] nsys-report-472f.sqlite [2/8] [======================90% ] nsys-report-472f.sqlite [2/8] [======================91% ] nsys-report-472f.sqlite [2/8] [======================92% ] nsys-report-472f.sqlite [2/8] [=======================93% ] nsys-report-472f.sqlite [2/8] [=======================94% ] nsys-report-472f.sqlite [2/8] [=======================95% ] nsys-report-472f.sqlite [2/8] [=======================96% ] nsys-report-472f.sqlite [2/8] [========================97% ] nsys-report-472f.sqlite [2/8] [========================98% ] nsys-report-472f.sqlite [2/8] [========================99% ] nsys-report-472f.sqlite [2/8] [========================100%] nsys-report-472f.sqlite [2/8] [========================100%] nsys-report-472f.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 65.5 1046221746 20 52311087.3 36977154.5 1343 144034477 47744624.9 poll + 25.3 403984081 582 694130.7 495337.5 1012 19219062 1320637.1 ioctl + 7.8 124190546 6 20698424.3 67121.0 7685 115338096 46492858.4 fread + 0.5 7610123 1 7610123.0 7610123.0 7610123 7610123 0.0 pthread_cond_wait + 0.4 7042952 1 7042952.0 7042952.0 7042952 7042952 0.0 pthread_cond_broadcast + 0.1 2350546 25 94021.8 9598.0 8175 1635631 323841.6 mmap64 + 0.1 1607267 38 42296.5 6812.5 3627 684381 126942.5 fopen + 0.1 872311 11 79301.0 27341.0 1052 372016 110728.5 write + 0.0 662102 32 20690.7 2364.0 1362 311784 71411.8 fclose + 0.0 599354 17 35256.1 4849.0 2225 375633 91066.5 mmap + 0.0 363971 2 181985.5 181985.5 164879 199092 24192.2 pthread_create + 0.0 355093 55 6456.2 6452.0 2405 12613 2601.2 open64 + 0.0 207257 1 207257.0 207257.0 207257 207257 0.0 fflush + 0.0 77074 1 77074.0 77074.0 77074 77074 0.0 fgets + 0.0 39155 8 4894.4 5225.0 2806 6762 1570.3 munmap + 0.0 36088 6 6014.7 5986.5 3056 8997 2029.8 open + 0.0 21803 3 7267.7 8657.0 3928 9218 2905.8 pipe2 + 0.0 18696 2 9348.0 9348.0 5150 13546 5936.9 socket + 0.0 17072 8 2134.0 1773.5 1132 3908 1047.0 read + 0.0 10660 1 10660.0 10660.0 10660 10660 0.0 connect + 0.0 6462 1 6462.0 6462.0 6462 6462 0.0 fopen64 + 0.0 3516 2 1758.0 1758.0 1412 2104 489.3 fwrite + 0.0 3275 3 1091.7 1082.0 1071 1122 26.8 fcntl + 0.0 1634 1 1634.0 1634.0 1634 1634 0.0 bind + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- --------------------------------- + 85.9 292488145 3 97496048.3 36522068.0 19767 255946310 138430374.4 cudaMemcpy + 5.8 19849256 147 135029.0 2775.0 2665 18870790 1556382.3 cudaLaunchKernel + 2.9 9971082 1 9971082.0 9971082.0 9971082 9971082 0.0 cudaGetDeviceProperties_v2_v12000 + 2.6 8726842 6 1454473.7 1517760.0 3446 2841288 1477680.0 cudaMalloc + 1.6 5319195 12 443266.3 5159.5 171 1876362 633355.6 cudaFree + 0.7 2439536 1 2439536.0 2439536.0 2439536 2439536 0.0 cudaFreeHost + 0.4 1329358 1 1329358.0 1329358.0 1329358 1329358 0.0 cudaMallocHost + 0.1 173678 838 207.3 190.0 80 1042 105.9 cuGetProcAddress_v2 + 0.0 35757 1 35757.0 35757.0 35757 35757 0.0 cudaMemset + 0.0 19408 18 1078.2 276.0 250 7765 1931.9 cudaEventCreateWithFlags + 0.0 12392 4 3098.0 2845.0 1002 5700 2009.4 cudaDeviceSynchronize + 0.0 7253 18 402.9 285.0 200 1854 379.7 cudaEventDestroy + 0.0 5671 4 1417.8 1428.0 1263 1552 141.7 cuInit + 0.0 1483 3 494.3 151.0 110 1222 630.5 cuModuleGetLoadingMode + 0.0 1262 2 631.0 631.0 461 801 240.4 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- --------- --------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 97.6 243397681 49 4967299.6 4539935.0 1227263 73746344 10132529.6 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 1.4 3576415 25 143056.6 142880.0 141472 146976 1318.1 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 0.8 1879295 12 156607.9 156656.0 155584 157120 383.9 void flash_attention_forward_kernel<(int)64, (int)64, (int)64>(const float *, const float *, const … + 0.1 228672 12 19056.0 19040.0 18880 19264 90.0 gelu_forward_kernel(float *, const float *, int) + 0.1 146208 24 6092.0 6080.0 5728 6560 355.1 residual_forward_kernel(float *, const float *, const float *, int) + 0.0 80768 1 80768.0 80768.0 80768 80768 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.0 78560 12 6546.7 6480.0 6400 6912 158.2 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.0 37952 12 3162.7 3152.0 3040 3296 108.2 unpermute_kernel(const float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 82.5 36437716 2 18218858.0 18218858.0 832 36436884 25764179.4 [CUDA memcpy Host-to-Device] + 16.5 7304990 1 7304990.0 7304990.0 7304990 7304990 0.0 [CUDA memcpy Device-to-Host] + 0.9 406816 1 406816.0 406816.0 406816 406816 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /tmp/nsys-report-d663.nsys-rep + /tmp/nsys-report-472f.sqlite diff --git a/output/perf/nsys/verify_op_12.err b/output/perf/nsys/verify_op_12.err new file mode 100644 index 0000000..0670358 --- /dev/null +++ b/output/perf/nsys/verify_op_12.err @@ -0,0 +1,9 @@ +WARNING: Device-side CUDA Event completion trace is currently enabled. + This may increase runtime overhead and the likelihood of false + dependencies across CUDA Streams. If you wish to avoid this, please + disable the feature with --cuda-event-trace=false. +Failed to create '/u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_12.nsys-rep': File exists. +Use `--force-overwrite true` to overwrite existing files. +Failed to create '/u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_12.sqlite': File exists. +Use `--force-overwrite true` to overwrite existing files. +SKIPPED: /tmp/nsys-report-9b06.sqlite does not contain NV Tools Extension (NVTX) data. diff --git a/output/perf/nsys/verify_op_12.out b/output/perf/nsys/verify_op_12.out new file mode 100644 index 0000000..bcf7568 --- /dev/null +++ b/output/perf/nsys/verify_op_12.out @@ -0,0 +1,109 @@ +job 18002391 starting on gpub073.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431793 +-39.836346, -39.836502 +-43.065910, -43.066078 +-42.828045, -42.828197 +-43.529541, -43.529701 +-44.318398, -44.318558 +-41.227425, -41.227592 +-41.270760, -41.270931 +-42.541393, -42.541584 +-42.394997, -42.395176 +---------------- Logits OK! (max err 0.000050, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-c6cc.qdstrm' + [1/8] [0% ] nsys-report-d644.nsys-rep [1/8] [0% ] nsys-report-d644.nsys-rep [1/8] [====26% ] nsys-report-d644.nsys-rep [1/8] [==========49% ] nsys-report-d644.nsys-rep [1/8] [===========53% ] nsys-report-d644.nsys-rep [1/8] [=======36% ] nsys-report-d644.nsys-rep [1/8] [===========52% ] nsys-report-d644.nsys-rep [1/8] [============54% ] nsys-report-d644.nsys-rep [1/8] [============55% ] nsys-report-d644.nsys-rep [1/8] [============56% ] nsys-report-d644.nsys-rep [1/8] [==============62% ] nsys-report-d644.nsys-rep [1/8] [==================78% ] nsys-report-d644.nsys-rep [1/8] [======================92% ] nsys-report-d644.nsys-rep [1/8] [========================100%] nsys-report-d644.nsys-rep [1/8] [========================100%] nsys-report-d644.nsys-rep + [2/8] [0% ] nsys-report-9b06.sqlite [2/8] [1% ] nsys-report-9b06.sqlite [2/8] [2% ] nsys-report-9b06.sqlite [2/8] [3% ] nsys-report-9b06.sqlite [2/8] [4% ] nsys-report-9b06.sqlite [2/8] [5% ] nsys-report-9b06.sqlite [2/8] [6% ] nsys-report-9b06.sqlite [2/8] [7% ] nsys-report-9b06.sqlite [2/8] [8% ] nsys-report-9b06.sqlite [2/8] [9% ] nsys-report-9b06.sqlite [2/8] [10% ] nsys-report-9b06.sqlite [2/8] [11% ] nsys-report-9b06.sqlite [2/8] [12% ] nsys-report-9b06.sqlite [2/8] [13% ] nsys-report-9b06.sqlite [2/8] [14% ] nsys-report-9b06.sqlite [2/8] [=15% ] nsys-report-9b06.sqlite [2/8] [=16% ] nsys-report-9b06.sqlite [2/8] [=17% ] nsys-report-9b06.sqlite [2/8] [==18% ] nsys-report-9b06.sqlite [2/8] [==19% ] nsys-report-9b06.sqlite [2/8] [==20% ] nsys-report-9b06.sqlite [2/8] [==21% ] nsys-report-9b06.sqlite [2/8] [===22% ] nsys-report-9b06.sqlite [2/8] [===23% ] nsys-report-9b06.sqlite [2/8] [===24% ] nsys-report-9b06.sqlite [2/8] [====25% ] nsys-report-9b06.sqlite [2/8] [====26% ] nsys-report-9b06.sqlite [2/8] [====27% ] nsys-report-9b06.sqlite [2/8] [====28% ] nsys-report-9b06.sqlite [2/8] [=====29% ] nsys-report-9b06.sqlite [2/8] [=====30% ] nsys-report-9b06.sqlite [2/8] [=====31% ] nsys-report-9b06.sqlite [2/8] [=====32% ] nsys-report-9b06.sqlite [2/8] [======33% ] nsys-report-9b06.sqlite [2/8] [======34% ] nsys-report-9b06.sqlite [2/8] [======35% ] nsys-report-9b06.sqlite [2/8] [=======36% ] nsys-report-9b06.sqlite [2/8] [=======37% ] nsys-report-9b06.sqlite [2/8] [=======38% ] nsys-report-9b06.sqlite [2/8] [=======39% ] nsys-report-9b06.sqlite [2/8] [========40% ] nsys-report-9b06.sqlite [2/8] [========41% ] nsys-report-9b06.sqlite [2/8] [========42% ] nsys-report-9b06.sqlite [2/8] [=========43% ] nsys-report-9b06.sqlite [2/8] [=========44% ] nsys-report-9b06.sqlite [2/8] [=========45% ] nsys-report-9b06.sqlite [2/8] [=========46% ] nsys-report-9b06.sqlite [2/8] [==========47% ] nsys-report-9b06.sqlite [2/8] [==========48% ] nsys-report-9b06.sqlite [2/8] [==========49% ] nsys-report-9b06.sqlite [2/8] [===========50% ] nsys-report-9b06.sqlite [2/8] [===========51% ] nsys-report-9b06.sqlite [2/8] [===========52% ] nsys-report-9b06.sqlite [2/8] [===========53% ] nsys-report-9b06.sqlite [2/8] [============54% ] nsys-report-9b06.sqlite [2/8] [============55% ] nsys-report-9b06.sqlite [2/8] [============56% ] nsys-report-9b06.sqlite [2/8] [============57% ] nsys-report-9b06.sqlite [2/8] [=============58% ] nsys-report-9b06.sqlite [2/8] [=============59% ] nsys-report-9b06.sqlite [2/8] [=============60% ] nsys-report-9b06.sqlite [2/8] [==============61% ] nsys-report-9b06.sqlite [2/8] [==============62% ] nsys-report-9b06.sqlite [2/8] [==============63% ] nsys-report-9b06.sqlite [2/8] [==============64% ] nsys-report-9b06.sqlite [2/8] [===============65% ] nsys-report-9b06.sqlite [2/8] [===============66% ] nsys-report-9b06.sqlite [2/8] [===============67% ] nsys-report-9b06.sqlite [2/8] [================68% ] nsys-report-9b06.sqlite [2/8] [================69% ] nsys-report-9b06.sqlite [2/8] [================70% ] nsys-report-9b06.sqlite [2/8] [================71% ] nsys-report-9b06.sqlite [2/8] [=================72% ] nsys-report-9b06.sqlite [2/8] [=================73% ] nsys-report-9b06.sqlite [2/8] [=================74% ] nsys-report-9b06.sqlite [2/8] [==================75% ] nsys-report-9b06.sqlite [2/8] [==================76% ] nsys-report-9b06.sqlite [2/8] [==================77% ] nsys-report-9b06.sqlite [2/8] [==================78% ] nsys-report-9b06.sqlite [2/8] [===================79% ] nsys-report-9b06.sqlite [2/8] [===================80% ] nsys-report-9b06.sqlite [2/8] [===================81% ] nsys-report-9b06.sqlite [2/8] [===================82% ] nsys-report-9b06.sqlite [2/8] [====================83% ] nsys-report-9b06.sqlite [2/8] [====================84% ] nsys-report-9b06.sqlite [2/8] [====================85% ] nsys-report-9b06.sqlite [2/8] [=====================86% ] nsys-report-9b06.sqlite [2/8] [=====================87% ] nsys-report-9b06.sqlite [2/8] [=====================88% ] nsys-report-9b06.sqlite [2/8] [=====================89% ] nsys-report-9b06.sqlite [2/8] [======================90% ] nsys-report-9b06.sqlite [2/8] [======================91% ] nsys-report-9b06.sqlite [2/8] [======================92% ] nsys-report-9b06.sqlite [2/8] [=======================93% ] nsys-report-9b06.sqlite [2/8] [=======================94% ] nsys-report-9b06.sqlite [2/8] [=======================95% ] nsys-report-9b06.sqlite [2/8] [=======================96% ] nsys-report-9b06.sqlite [2/8] [========================97% ] nsys-report-9b06.sqlite [2/8] [========================98% ] nsys-report-9b06.sqlite [2/8] [========================99% ] nsys-report-9b06.sqlite [2/8] [========================100%] nsys-report-9b06.sqlite [2/8] [========================100%] nsys-report-9b06.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 61.1 737068265 17 43356956.8 31803265.0 2204 136058155 46033325.4 poll + 29.3 352852419 581 607319.1 280456.0 1082 20080933 1246338.6 ioctl + 7.2 86629708 6 14438284.7 75346.0 5560 78483379 31535650.3 fread + 1.1 13589855 1 13589855.0 13589855.0 13589855 13589855 0.0 pthread_cond_wait + 0.6 7448723 1 7448723.0 7448723.0 7448723 7448723 0.0 pthread_cond_broadcast + 0.2 2347161 25 93886.4 9638.0 7675 1629425 322632.7 mmap64 + 0.2 2110733 32 65960.4 2189.0 1352 1703774 304577.4 fclose + 0.1 1693364 38 44562.2 6256.0 3587 809497 144162.0 fopen + 0.1 746287 11 67844.3 27722.0 1132 318406 94484.9 write + 0.0 594968 17 34998.1 4739.0 2575 402164 96189.6 mmap + 0.0 361558 2 180779.0 180779.0 164709 196849 22726.4 pthread_create + 0.0 353489 55 6427.1 6332.0 2344 13415 2698.2 open64 + 0.0 221045 1 221045.0 221045.0 221045 221045 0.0 fflush + 0.0 76323 1 76323.0 76323.0 76323 76323 0.0 fgets + 0.0 37250 8 4656.3 4112.5 2875 7805 1734.3 munmap + 0.0 36069 6 6011.5 5996.5 3086 9027 2010.2 open + 0.0 20939 3 6979.7 8526.0 3727 8686 2818.0 pipe2 + 0.0 18773 9 2085.9 2083.0 1082 3727 979.8 read + 0.0 17263 2 8631.5 8631.5 6523 10740 2981.9 socket + 0.0 9477 1 9477.0 9477.0 9477 9477 0.0 connect + 0.0 6272 1 6272.0 6272.0 6272 6272 0.0 fopen64 + 0.0 4249 2 2124.5 2124.5 1413 2836 1006.2 fwrite + 0.0 2134 1 2134.0 2134.0 2134 2134 0.0 bind + 0.0 2054 2 1027.0 1027.0 1012 1042 21.2 fcntl + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- -------- ----------- --------------------------------- + 76.2 121358249 3 40452749.7 58133092.0 22181 63202976 35105542.2 cudaMemcpy + 9.7 15479869 6 2579978.2 1263944.5 4198 8351576 3362372.0 cudaMalloc + 6.3 10045470 1 10045470.0 10045470.0 10045470 10045470 0.0 cudaGetDeviceProperties_v2_v12000 + 3.2 5100454 12 425037.8 6347.5 180 1804402 614184.2 cudaFree + 1.9 2948206 1 2948206.0 2948206.0 2948206 2948206 0.0 cudaMallocHost + 1.6 2474178 1 2474178.0 2474178.0 2474178 2474178 0.0 cudaFreeHost + 1.1 1675156 171 9796.2 2805.0 2675 1050069 80118.8 cudaLaunchKernel + 0.1 154831 838 184.8 150.0 80 762 92.5 cuGetProcAddress_v2 + 0.0 39014 1 39014.0 39014.0 39014 39014 0.0 cudaMemset + 0.0 17903 18 994.6 296.0 250 6583 1655.0 cudaEventCreateWithFlags + 0.0 12222 4 3055.5 2619.5 982 6001 2113.3 cudaDeviceSynchronize + 0.0 6803 18 377.9 251.0 210 2014 418.1 cudaEventDestroy + 0.0 5841 4 1460.3 1362.5 1182 1934 328.0 cuInit + 0.0 1352 3 450.7 120.0 90 1142 598.9 cuModuleGetLoadingMode + 0.0 852 2 426.0 426.0 301 551 176.8 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- -------- -------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 88.5 45226826 49 922996.4 851682.0 243969 12757309 1750034.1 matmul_kernel(const float *, float *, const float *, const float *, int, int, int, int) + 7.0 3578022 25 143120.9 143072.0 141120 146912 1197.9 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 2.3 1153027 12 96085.6 96112.5 95904 96192 94.9 preattention_scores_kernel(const float *, const float *, int, int, int, int, float *) + 0.7 346208 12 28850.7 28688.0 28352 29760 446.4 attention_v_kernel(float *, const float *, const float *, int, int, int, int) + 0.4 228480 12 19040.0 19040.0 18880 19232 79.6 gelu_forward_kernel(float *, const float *, int) + 0.4 210304 12 17525.3 17520.0 17344 17792 116.8 softmax_forward_kernel(float *, float, const float *, int, int) + 0.3 144992 24 6041.3 6048.0 5664 6464 367.5 residual_forward_kernel(float *, const float *, const float *, int) + 0.2 84320 12 7026.7 7024.0 6880 7168 92.0 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.2 80545 1 80545.0 80545.0 80545 80545 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.1 70400 12 5866.7 5872.0 5792 5888 28.4 unpermute_kernel(const float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 88.9 63114351 2 31557175.5 31557175.5 832 63113519 44627409.0 [CUDA memcpy Host-to-Device] + 10.5 7436977 1 7436977.0 7436977.0 7436977 7436977 0.0 [CUDA memcpy Device-to-Host] + 0.6 407969 1 407969.0 407969.0 407969 407969 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /tmp/nsys-report-d644.nsys-rep + /tmp/nsys-report-9b06.sqlite diff --git a/output/perf/nsys/verify_op_13.err b/output/perf/nsys/verify_op_13.err new file mode 100644 index 0000000..809b8b5 --- /dev/null +++ b/output/perf/nsys/verify_op_13.err @@ -0,0 +1,5 @@ +WARNING: Device-side CUDA Event completion trace is currently enabled. + This may increase runtime overhead and the likelihood of false + dependencies across CUDA Streams. If you wish to avoid this, please + disable the feature with --cuda-event-trace=false. +SKIPPED: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_13.sqlite does not contain NV Tools Extension (NVTX) data. diff --git a/output/perf/nsys/verify_op_13.out b/output/perf/nsys/verify_op_13.out new file mode 100644 index 0000000..530be62 --- /dev/null +++ b/output/perf/nsys/verify_op_13.out @@ -0,0 +1,108 @@ +job 18003870 starting on gpub055.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-357a.qdstrm' + [1/8] [0% ] report_op_13.nsys-rep [1/8] [0% ] report_op_13.nsys-rep [1/8] [====26% ] report_op_13.nsys-rep [1/8] [==========49% ] report_op_13.nsys-rep [1/8] [===========50% ] report_op_13.nsys-rep [1/8] [===========51% ] report_op_13.nsys-rep [1/8] [=======38% ] report_op_13.nsys-rep [1/8] [===========51% ] report_op_13.nsys-rep [1/8] [===========52% ] report_op_13.nsys-rep [1/8] [===========53% ] report_op_13.nsys-rep [1/8] [==============64% ] report_op_13.nsys-rep [1/8] [======================90% ] report_op_13.nsys-rep [1/8] [========================100%] report_op_13.nsys-rep [1/8] [========================100%] report_op_13.nsys-rep + [2/8] [0% ] report_op_13.sqlite [2/8] [1% ] report_op_13.sqlite [2/8] [2% ] report_op_13.sqlite [2/8] [3% ] report_op_13.sqlite [2/8] [4% ] report_op_13.sqlite [2/8] [5% ] report_op_13.sqlite [2/8] [6% ] report_op_13.sqlite [2/8] [7% ] report_op_13.sqlite [2/8] [8% ] report_op_13.sqlite [2/8] [9% ] report_op_13.sqlite [2/8] [10% ] report_op_13.sqlite [2/8] [11% ] report_op_13.sqlite [2/8] [12% ] report_op_13.sqlite [2/8] [13% ] report_op_13.sqlite [2/8] [14% ] report_op_13.sqlite [2/8] [=15% ] report_op_13.sqlite [2/8] [=16% ] report_op_13.sqlite [2/8] [=17% ] report_op_13.sqlite [2/8] [==18% ] report_op_13.sqlite [2/8] [==19% ] report_op_13.sqlite [2/8] [==20% ] report_op_13.sqlite [2/8] [==21% ] report_op_13.sqlite [2/8] [===22% ] report_op_13.sqlite [2/8] [===23% ] report_op_13.sqlite [2/8] [===24% ] report_op_13.sqlite [2/8] [====25% ] report_op_13.sqlite [2/8] [====26% ] report_op_13.sqlite [2/8] [====27% ] report_op_13.sqlite [2/8] [====28% ] report_op_13.sqlite [2/8] [=====29% ] report_op_13.sqlite [2/8] [=====30% ] report_op_13.sqlite [2/8] [=====31% ] report_op_13.sqlite [2/8] [=====32% ] report_op_13.sqlite [2/8] [======33% ] report_op_13.sqlite [2/8] [======34% ] report_op_13.sqlite [2/8] [======35% ] report_op_13.sqlite [2/8] [=======36% ] report_op_13.sqlite [2/8] [=======37% ] report_op_13.sqlite [2/8] [=======38% ] report_op_13.sqlite [2/8] [=======39% ] report_op_13.sqlite [2/8] [========40% ] report_op_13.sqlite [2/8] [========41% ] report_op_13.sqlite [2/8] [========42% ] report_op_13.sqlite [2/8] [=========43% ] report_op_13.sqlite [2/8] [=========44% ] report_op_13.sqlite [2/8] [=========45% ] report_op_13.sqlite [2/8] [=========46% ] report_op_13.sqlite [2/8] [==========47% ] report_op_13.sqlite [2/8] [==========48% ] report_op_13.sqlite [2/8] [==========49% ] report_op_13.sqlite [2/8] [===========50% ] report_op_13.sqlite [2/8] [===========51% ] report_op_13.sqlite [2/8] [===========52% ] report_op_13.sqlite [2/8] [===========53% ] report_op_13.sqlite [2/8] [============54% ] report_op_13.sqlite [2/8] [============55% ] report_op_13.sqlite [2/8] [============56% ] report_op_13.sqlite [2/8] [============57% ] report_op_13.sqlite [2/8] [=============58% ] report_op_13.sqlite [2/8] [=============59% ] report_op_13.sqlite [2/8] [=============60% ] report_op_13.sqlite [2/8] [==============61% ] report_op_13.sqlite [2/8] [==============62% ] report_op_13.sqlite [2/8] [==============63% ] report_op_13.sqlite [2/8] [==============64% ] report_op_13.sqlite [2/8] [===============65% ] report_op_13.sqlite [2/8] [===============66% ] report_op_13.sqlite [2/8] [===============67% ] report_op_13.sqlite [2/8] [================68% ] report_op_13.sqlite [2/8] [================69% ] report_op_13.sqlite [2/8] [================70% ] report_op_13.sqlite [2/8] [================71% ] report_op_13.sqlite [2/8] [=================72% ] report_op_13.sqlite [2/8] [=================73% ] report_op_13.sqlite [2/8] [=================74% ] report_op_13.sqlite [2/8] [==================75% ] report_op_13.sqlite [2/8] [==================76% ] report_op_13.sqlite [2/8] [==================77% ] report_op_13.sqlite [2/8] [==================78% ] report_op_13.sqlite [2/8] [===================79% ] report_op_13.sqlite [2/8] [===================80% ] report_op_13.sqlite [2/8] [===================81% ] report_op_13.sqlite [2/8] [===================82% ] report_op_13.sqlite [2/8] [====================83% ] report_op_13.sqlite [2/8] [====================84% ] report_op_13.sqlite [2/8] [====================85% ] report_op_13.sqlite [2/8] [=====================86% ] report_op_13.sqlite [2/8] [=====================87% ] report_op_13.sqlite [2/8] [=====================88% ] report_op_13.sqlite [2/8] [=====================89% ] report_op_13.sqlite [2/8] [======================90% ] report_op_13.sqlite [2/8] [======================91% ] report_op_13.sqlite [2/8] [======================92% ] report_op_13.sqlite [2/8] [=======================93% ] report_op_13.sqlite [2/8] [=======================94% ] report_op_13.sqlite [2/8] [=======================95% ] report_op_13.sqlite [2/8] [=======================96% ] report_op_13.sqlite [2/8] [========================97% ] report_op_13.sqlite [2/8] [========================98% ] report_op_13.sqlite [2/8] [========================99% ] report_op_13.sqlite [2/8] [========================100%] report_op_13.sqlite [2/8] [========================100%] report_op_13.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 62.3 1192177975 22 54189908.0 60718799.0 1293 100135381 46848153.7 poll + 27.8 532636664 6 88772777.3 616568.5 7834 482699408 193951324.3 fread + 7.6 145642532 586 248536.7 21555.0 1032 28349154 1447180.4 ioctl + 1.1 20477585 1 20477585.0 20477585.0 20477585 20477585 0.0 pthread_cond_wait + 0.5 9619823 38 253153.2 6146.0 3567 8444540 1368023.7 fopen + 0.4 7231291 1 7231291.0 7231291.0 7231291 7231291 0.0 pthread_cond_broadcast + 0.1 2321011 25 92840.4 9037.0 7353 1601907 317206.9 mmap64 + 0.1 1381765 10 138176.5 29961.0 22552 501578 193339.9 write + 0.0 665856 32 20808.0 2144.0 1353 313456 72473.1 fclose + 0.0 574715 17 33806.8 4579.0 1844 383638 91786.3 mmap + 0.0 347480 2 173740.0 173740.0 156984 190496 23696.6 pthread_create + 0.0 313434 55 5698.8 5871.0 2334 9788 2009.1 open64 + 0.0 206186 1 206186.0 206186.0 206186 206186 0.0 fflush + 0.0 77074 1 77074.0 77074.0 77074 77074 0.0 fgets + 0.0 48280 8 6035.0 4824.0 2665 12333 3569.5 munmap + 0.0 35765 6 5960.8 6016.0 3055 8936 1976.5 open + 0.0 23155 3 7718.3 9318.0 3457 10380 3728.4 pipe2 + 0.0 17893 9 1988.1 1463.0 1012 3617 1022.1 read + 0.0 15579 2 7789.5 7789.5 5791 9788 2826.3 socket + 0.0 9318 1 9318.0 9318.0 9318 9318 0.0 connect + 0.0 6342 1 6342.0 6342.0 6342 6342 0.0 fopen64 + 0.0 3918 2 1959.0 1959.0 1453 2465 715.6 fwrite + 0.0 1623 1 1623.0 1623.0 1623 1623 0.0 bind + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- --------------------------------- + 86.7 292338879 3 97446293.0 36179585.0 18745 256140549 138617465.5 cudaMemcpy + 10.4 34929986 12 2910832.2 8676.5 230 29248496 8331465.4 cudaFree + 1.1 3690547 171 21582.1 2875.0 2735 3041560 232340.0 cudaLaunchKernel + 0.7 2477256 1 2477256.0 2477256.0 2477256 2477256 0.0 cudaFreeHost + 0.4 1441077 1 1441077.0 1441077.0 1441077 1441077 0.0 cudaGetDeviceProperties_v2_v12000 + 0.4 1293189 1 1293189.0 1293189.0 1293189 1293189 0.0 cudaMallocHost + 0.2 708847 6 118141.2 115361.0 3406 282809 107158.9 cudaMalloc + 0.1 176229 838 210.3 180.0 89 5641 213.4 cuGetProcAddress_v2 + 0.0 34836 1 34836.0 34836.0 34836 34836 0.0 cudaMemset + 0.0 16491 4 4122.8 3136.5 1021 9197 3611.3 cudaDeviceSynchronize + 0.0 11905 18 661.4 280.5 250 3005 821.3 cudaEventCreateWithFlags + 0.0 7888 18 438.2 270.5 210 2675 575.7 cudaEventDestroy + 0.0 5102 4 1275.5 1278.0 1233 1313 33.0 cuInit + 0.0 1753 3 584.3 461.0 110 1182 546.5 cuModuleGetLoadingMode + 0.0 993 2 496.5 496.5 331 662 234.1 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- --------- --------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 97.6 243490921 49 4969202.5 4539029.0 1227366 73795414 10139382.4 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 1.5 3693489 25 147739.6 148096.0 144832 149601 1394.9 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 0.5 1145094 12 95424.5 95232.5 95073 96065 370.8 preattention_scores_kernel(const float *, const float *, int, int, int, int, float *) + 0.1 334432 12 27869.3 27968.0 26880 28704 511.5 attention_v_kernel(float *, const float *, const float *, int, int, int, int) + 0.1 227360 12 18946.7 18976.0 18720 19008 74.1 gelu_forward_kernel(float *, const float *, int) + 0.1 207585 12 17298.8 17344.0 17120 17440 111.3 softmax_forward_kernel(float *, float, const float *, int, int) + 0.1 145664 24 6069.3 6128.0 5664 6464 351.1 residual_forward_kernel(float *, const float *, const float *, int) + 0.0 96320 12 8026.7 8016.0 7840 8224 115.6 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.0 80128 1 80128.0 80128.0 80128 80128 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.0 70530 12 5877.5 5888.0 5856 5920 20.8 unpermute_kernel(const float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 82.8 36065319 2 18032659.5 18032659.5 832 36064487 25500855.0 [CUDA memcpy Host-to-Device] + 16.3 7099265 1 7099265.0 7099265.0 7099265 7099265 0.0 [CUDA memcpy Device-to-Host] + 0.9 407490 1 407490.0 407490.0 407490 407490 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.393 1 232.393 232.393 232.393 232.393 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_13.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_13.sqlite diff --git a/output/perf/nsys/verify_op_16.err b/output/perf/nsys/verify_op_16.err new file mode 100644 index 0000000..855e3e1 --- /dev/null +++ b/output/perf/nsys/verify_op_16.err @@ -0,0 +1,5 @@ +WARNING: Device-side CUDA Event completion trace is currently enabled. + This may increase runtime overhead and the likelihood of false + dependencies across CUDA Streams. If you wish to avoid this, please + disable the feature with --cuda-event-trace=false. +SKIPPED: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_op_16.sqlite does not contain NV Tools Extension (NVTX) data. diff --git a/output/perf/nsys/verify_op_16.out b/output/perf/nsys/verify_op_16.out new file mode 100644 index 0000000..2534a8f --- /dev/null +++ b/output/perf/nsys/verify_op_16.out @@ -0,0 +1,116 @@ +job 18004586 starting on gpub041.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.315117 +-39.836346, -39.732384 +-43.065910, -42.960098 +-42.828045, -42.723515 +-43.529541, -43.418613 +-44.318398, -44.210033 +-41.227425, -41.119377 +-41.270760, -41.164490 +-42.541393, -42.428894 +-42.394997, -42.288929 +---------------- Logits OK! (max err 0.082334, RMSE 0.002728) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-cacf.qdstrm' + [1/8] [0% ] report_op_16.nsys-rep [1/8] [0% ] report_op_16.nsys-rep [1/8] [====27% ] report_op_16.nsys-rep [1/8] [====28% ] report_op_16.nsys-rep [1/8] [=====29% ] report_op_16.nsys-rep [1/8] [=====30% ] report_op_16.nsys-rep [1/8] [=======38% ] report_op_16.nsys-rep [1/8] [=========46% ] report_op_16.nsys-rep [1/8] [==========47% ] report_op_16.nsys-rep [1/8] [==========48% ] report_op_16.nsys-rep [1/8] [==========49% ] report_op_16.nsys-rep [1/8] [============55% ] report_op_16.nsys-rep [1/8] [============56% ] report_op_16.nsys-rep [1/8] [============57% ] report_op_16.nsys-rep [1/8] [=============58% ] report_op_16.nsys-rep [1/8] [==================77% ] report_op_16.nsys-rep [1/8] [=======================94% ] report_op_16.nsys-rep [1/8] [========================100%] report_op_16.nsys-rep [1/8] [========================100%] report_op_16.nsys-rep + [2/8] [0% ] report_op_16.sqlite [2/8] [1% ] report_op_16.sqlite [2/8] [2% ] report_op_16.sqlite [2/8] [3% ] report_op_16.sqlite [2/8] [4% ] report_op_16.sqlite [2/8] [5% ] report_op_16.sqlite [2/8] [6% ] report_op_16.sqlite [2/8] [7% ] report_op_16.sqlite [2/8] [8% ] report_op_16.sqlite [2/8] [9% ] report_op_16.sqlite [2/8] [10% ] report_op_16.sqlite [2/8] [11% ] report_op_16.sqlite [2/8] [12% ] report_op_16.sqlite [2/8] [13% ] report_op_16.sqlite [2/8] [14% ] report_op_16.sqlite [2/8] [=15% ] report_op_16.sqlite [2/8] [=16% ] report_op_16.sqlite [2/8] [=17% ] report_op_16.sqlite [2/8] [==18% ] report_op_16.sqlite [2/8] [==19% ] report_op_16.sqlite [2/8] [==20% ] report_op_16.sqlite [2/8] [==21% ] report_op_16.sqlite [2/8] [===22% ] report_op_16.sqlite [2/8] [===23% ] report_op_16.sqlite [2/8] [===24% ] report_op_16.sqlite [2/8] [====25% ] report_op_16.sqlite [2/8] [====26% ] report_op_16.sqlite [2/8] [====27% ] report_op_16.sqlite [2/8] [====28% ] report_op_16.sqlite [2/8] [=====29% ] report_op_16.sqlite [2/8] [=====30% ] report_op_16.sqlite [2/8] [=====31% ] report_op_16.sqlite [2/8] [=====32% ] report_op_16.sqlite [2/8] [======33% ] report_op_16.sqlite [2/8] [======34% ] report_op_16.sqlite [2/8] [======35% ] report_op_16.sqlite [2/8] [=======36% ] report_op_16.sqlite [2/8] [=======37% ] report_op_16.sqlite [2/8] [=======38% ] report_op_16.sqlite [2/8] [=======39% ] report_op_16.sqlite [2/8] [========40% ] report_op_16.sqlite [2/8] [========41% ] report_op_16.sqlite [2/8] [========42% ] report_op_16.sqlite [2/8] [=========43% ] report_op_16.sqlite [2/8] [=========44% ] report_op_16.sqlite [2/8] [=========45% ] report_op_16.sqlite [2/8] [=========46% ] report_op_16.sqlite [2/8] [==========47% ] report_op_16.sqlite [2/8] [==========48% ] report_op_16.sqlite [2/8] [==========49% ] report_op_16.sqlite [2/8] [===========50% ] report_op_16.sqlite [2/8] [===========51% ] report_op_16.sqlite [2/8] [===========52% ] report_op_16.sqlite [2/8] [===========53% ] report_op_16.sqlite [2/8] [============54% ] report_op_16.sqlite [2/8] [============55% ] report_op_16.sqlite [2/8] [============56% ] report_op_16.sqlite [2/8] [============57% ] report_op_16.sqlite [2/8] [=============58% ] report_op_16.sqlite [2/8] [=============59% ] report_op_16.sqlite [2/8] [=============60% ] report_op_16.sqlite [2/8] [==============61% ] report_op_16.sqlite [2/8] [==============62% ] report_op_16.sqlite [2/8] [==============63% ] report_op_16.sqlite [2/8] [==============64% ] report_op_16.sqlite [2/8] [===============65% ] report_op_16.sqlite [2/8] [===============66% ] report_op_16.sqlite [2/8] [===============67% ] report_op_16.sqlite [2/8] [================68% ] report_op_16.sqlite [2/8] [================69% ] report_op_16.sqlite [2/8] [================70% ] report_op_16.sqlite [2/8] [================71% ] report_op_16.sqlite [2/8] [=================72% ] report_op_16.sqlite [2/8] [=================73% ] report_op_16.sqlite [2/8] [=================74% ] report_op_16.sqlite [2/8] [==================75% ] report_op_16.sqlite [2/8] [==================76% ] report_op_16.sqlite [2/8] [==================77% ] report_op_16.sqlite [2/8] [==================78% ] report_op_16.sqlite [2/8] [===================79% ] report_op_16.sqlite [2/8] [===================80% ] report_op_16.sqlite [2/8] [===================81% ] report_op_16.sqlite [2/8] [===================82% ] report_op_16.sqlite [2/8] [====================83% ] report_op_16.sqlite [2/8] [====================84% ] report_op_16.sqlite [2/8] [====================85% ] report_op_16.sqlite [2/8] [=====================86% ] report_op_16.sqlite [2/8] [=====================87% ] report_op_16.sqlite [2/8] [=====================88% ] report_op_16.sqlite [2/8] [=====================89% ] report_op_16.sqlite [2/8] [======================90% ] report_op_16.sqlite [2/8] [======================91% ] report_op_16.sqlite [2/8] [======================92% ] report_op_16.sqlite [2/8] [=======================93% ] report_op_16.sqlite [2/8] [=======================94% ] report_op_16.sqlite [2/8] [=======================95% ] report_op_16.sqlite [2/8] [=======================96% ] report_op_16.sqlite [2/8] [========================97% ] report_op_16.sqlite [2/8] [========================98% ] report_op_16.sqlite [2/8] [========================99% ] report_op_16.sqlite [2/8] [========================100%] report_op_16.sqlite [2/8] [========================100%] report_op_16.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ----------- ---------- -------- --------- ----------- -------------- + 58.3 1107473464 21 52736831.6 32614355.0 1322 102246365 47278143.8 poll + 32.4 615776283 6 102629380.5 9779477.0 6893 526459513 209303008.5 fread + 8.7 164992616 592 278703.7 23008.0 1041 22196543 1294110.2 ioctl + 0.2 4255919 38 111997.9 6477.0 3637 3134408 512054.3 fopen + 0.1 2359811 25 94392.4 8766.0 7474 1634199 323598.5 mmap64 + 0.1 999815 32 31244.2 2119.0 1333 525704 114089.3 fclose + 0.0 836867 11 76078.8 29566.0 1603 467756 132907.7 write + 0.0 568205 17 33423.8 8155.0 1943 376916 90182.7 mmap + 0.0 345457 2 172728.5 172728.5 163176 182281 13509.3 pthread_create + 0.0 328243 55 5968.1 5862.0 2344 13786 2471.7 open64 + 0.0 223538 1 223538.0 223538.0 223538 223538 0.0 fflush + 0.0 80441 1 80441.0 80441.0 80441 80441 0.0 fgets + 0.0 46428 9 5158.7 4038.0 2575 9789 2506.9 munmap + 0.0 37720 6 6286.7 6542.5 3276 8926 1948.3 open + 0.0 27181 3 9060.3 8235.0 6613 12333 2948.0 pipe2 + 0.0 19594 11 1781.3 1282.0 1002 3506 893.6 read + 0.0 17683 2 8841.5 8841.5 5851 11832 4229.2 socket + 0.0 10981 1 10981.0 10981.0 10981 10981 0.0 connect + 0.0 6862 1 6862.0 6862.0 6862 6862 0.0 fopen64 + 0.0 4558 2 2279.0 2279.0 2074 2484 289.9 fwrite + 0.0 2174 1 2174.0 2174.0 2174 2174 0.0 bind + 0.0 1062 1 1062.0 1062.0 1062 1062 0.0 listen + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- -------- ----------- --------------------------------- + 54.2 51711904 3 17237301.3 15706111.0 19487 35986306 18032232.9 cudaMemcpy + 22.1 21075926 12 1756327.2 15158.5 190 16493445 4691526.2 cudaFree + 10.2 9764437 4 2441109.3 2908985.5 877652 3068814 1045305.5 cuLibraryLoadData + 3.8 3644594 207 17606.7 4519.0 2725 2467218 171275.3 cudaLaunchKernel + 2.5 2409812 1 2409812.0 2409812.0 2409812 2409812 0.0 cudaFreeHost + 1.7 1608458 18 89358.8 305.5 260 1597961 376498.2 cudaEventCreateWithFlags + 1.4 1351740 1 1351740.0 1351740.0 1351740 1351740 0.0 cudaMallocHost + 1.4 1315042 1 1315042.0 1315042.0 1315042 1315042 0.0 cudaGetDeviceProperties_v2_v12000 + 1.1 1038424 6 173070.7 135794.0 4840 445143 174111.4 cudaMalloc + 1.0 977961 4 244490.3 219145.5 1072 538598 283164.8 cuLibraryGetKernel + 0.2 229567 838 273.9 180.0 90 48441 1670.5 cuGetProcAddress_v2 + 0.1 137239 36 3812.2 3146.0 2094 14667 2165.3 cudaMemsetAsync + 0.1 96319 36 2675.5 2049.0 1833 7945 1134.7 cudaEventRecord + 0.0 44693 1 44693.0 44693.0 44693 44693 0.0 cudaMemset + 0.0 38138 180 211.9 130.0 70 12083 894.1 cuStreamGetCaptureInfo_v2 + 0.0 21040 4 5260.0 3141.0 1503 13255 5424.7 cudaDeviceSynchronize + 0.0 9043 18 502.4 275.5 200 3386 741.5 cudaEventDestroy + 0.0 5723 4 1430.8 1443.5 1163 1673 221.4 cuInit + 0.0 3677 1 3677.0 3677.0 3677 3677 0.0 cudaEventQuery + 0.0 1714 3 571.3 140.0 111 1463 772.3 cuModuleGetLoadingMode + 0.0 941 2 470.5 470.5 300 641 241.1 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- -------- -------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 35.8 3686110 25 147444.4 147616.0 145600 149216 1035.4 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 16.2 1663328 13 127948.3 59392.0 58912 950752 247221.8 ampere_sgemm_128x64_tn + 14.9 1536224 12 128018.7 128112.0 125600 129472 1083.3 ampere_sgemm_64x64_tn + 11.1 1146944 12 95578.7 95616.0 94720 95904 297.0 preattention_scores_kernel(const float *, const float *, int, int, int, int, float *) + 6.5 670624 12 55885.3 55840.0 55136 56928 530.5 void cutlass::Kernel(const float *, const float *, const … + 0.1 227840 12 18986.7 19008.0 18688 19136 115.2 gelu_forward_kernel(float *, const float *, int) + 0.1 146144 24 6089.3 6128.0 5696 6464 346.6 residual_forward_kernel(float *, const float *, const float *, int) + 0.0 79776 12 6648.0 6640.0 6464 6816 119.1 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.0 79296 1 79296.0 79296.0 79296 79296 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.0 36640 12 3053.3 3040.0 3040 3104 21.4 unpermute_kernel(const float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 82.7 36306187 2 18153093.5 18153093.5 832 36305355 25671174.4 [CUDA memcpy Host-to-Device] + 16.4 7213212 1 7213212.0 7213212.0 7213212 7213212 0.0 [CUDA memcpy Device-to-Host] + 0.9 407200 1 407200.0 407200.0 407200 407200 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_req_4.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/report_req_4.sqlite diff --git a/output/simple/verify_op_11.err b/output/simple/verify_op_11.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_11.out b/output/simple/verify_op_11.out new file mode 100644 index 0000000..0998750 --- /dev/null +++ b/output/simple/verify_op_11.out @@ -0,0 +1,21 @@ +job 18002664 is starting on gpub022.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_op_12.err b/output/simple/verify_op_12.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_12.out b/output/simple/verify_op_12.out new file mode 100644 index 0000000..9e983cd --- /dev/null +++ b/output/simple/verify_op_12.out @@ -0,0 +1,21 @@ +job 18002591 is starting on gpub089.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431793 +-39.836346, -39.836502 +-43.065910, -43.066078 +-42.828045, -42.828197 +-43.529541, -43.529701 +-44.318398, -44.318558 +-41.227425, -41.227592 +-41.270760, -41.270931 +-42.541393, -42.541584 +-42.394997, -42.395176 +---------------- Logits OK! (max err 0.000050, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_op_13.err b/output/simple/verify_op_13.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_13.out b/output/simple/verify_op_13.out new file mode 100644 index 0000000..548edc8 --- /dev/null +++ b/output/simple/verify_op_13.out @@ -0,0 +1,21 @@ +job 18003856 is starting on gpub022.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_op_16.err b/output/simple/verify_op_16.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_16.out b/output/simple/verify_op_16.out new file mode 100644 index 0000000..e35a2f8 --- /dev/null +++ b/output/simple/verify_op_16.out @@ -0,0 +1,21 @@ +job 18004581 is starting on gpub035.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.315117 +-39.836346, -39.732384 +-43.065910, -42.960098 +-42.828045, -42.723515 +-43.529541, -43.418613 +-44.318398, -44.210033 +-41.227425, -41.119377 +-41.270760, -41.164490 +-42.541393, -42.428894 +-42.394997, -42.288929 +---------------- Logits OK! (max err 0.082334, RMSE 0.002728) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_op_8.err b/output/simple/verify_op_8.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_8.out b/output/simple/verify_op_8.out new file mode 100644 index 0000000..3c7c843 --- /dev/null +++ b/output/simple/verify_op_8.out @@ -0,0 +1,21 @@ +job 17968509 is starting on gpub005.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_op_9.err b/output/simple/verify_op_9.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_op_9.out b/output/simple/verify_op_9.out new file mode 100644 index 0000000..cff5c08 --- /dev/null +++ b/output/simple/verify_op_9.out @@ -0,0 +1,21 @@ +job 17968382 is starting on gpub003.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output/simple/verify_req_4.err b/output/simple/verify_req_4.err new file mode 100644 index 0000000..e69de29 diff --git a/output/simple/verify_req_4.out b/output/simple/verify_req_4.out new file mode 100644 index 0000000..fb99dc4 --- /dev/null +++ b/output/simple/verify_req_4.out @@ -0,0 +1,21 @@ +job 17971913 is starting on gpub022.delta.ncsa.illinois.edu + + +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ diff --git a/output_verification_rand.cu b/output_verification_rand.cu new file mode 100644 index 0000000..fd53731 --- /dev/null +++ b/output_verification_rand.cu @@ -0,0 +1,498 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Include CPU kernels +#include "cpu_kernels/attention.cuh" +#include "cpu_kernels/encoder.cuh" +#include "cpu_kernels/gelu.cuh" +#include "cpu_kernels/layernorm.cuh" +#include "cpu_kernels/matmul.cuh" +#include "cpu_kernels/residual.cuh" + +// Include GPU kernels +#include "kernels/attention.cuh" +#include "kernels/encoder.cuh" +#include "kernels/gelu.cuh" +#include "kernels/layernorm.cuh" +#include "kernels/matmul.cuh" +#include "kernels/residual.cuh" +#include "kernels/softmax.cuh" + +// Include CUDA utilities +#include "utils/cuda_utils.cuh" + +int main() { + + // set up the device + int deviceIdx = 0; + cudaCheck(cudaSetDevice(deviceIdx)); + cudaDeviceProp deviceProp; + cudaGetDeviceProperties(&deviceProp, deviceIdx); + + // setup cuBLAS and cuBLASLt + cublasCheck(cublasCreate(&cublas_handle)); + // TF32 precision is equivalent to torch.set_float32_matmul_precision('high') + int enable_tf32 = 0; + cublas_compute_type = enable_tf32 ? CUBLAS_COMPUTE_32F_FAST_TF32 : CUBLAS_COMPUTE_32F; + cublasMath_t cublas_math_mode = enable_tf32 ? CUBLAS_TF32_TENSOR_OP_MATH : CUBLAS_DEFAULT_MATH; + cublasCheck(cublasSetMathMode(cublas_handle, cublas_math_mode)); + + const int B = 4; // Batch size + const int T = 128; // Sequence length + const int C = 128; // Input channels + const int OC = 4 * C; // Output channels + + const int NH = 12; // Number of heads + + const float error_threshold = 0; // TODO: Set the error threshold for GPU vs CPU output comparisons + // This is in unit of percentage difference in output values + // Change this to a small value (like 0.01) for your own testing + + const float ep = 0.0001; // Epsilon value for percentage calculation to avoid getting infinity + + // used in multi-head self-attention + const int HS = C / NH; // head size + + // Generate random input data + srand(time(NULL)); + + /*------------------------ encoder_forward test ------------------------*/ + + // Allocate memory for input data + int* enc_inp_cpu = (int*)malloc(B * T * sizeof(int)); + float* wte_enc = (float*)malloc(C * C * sizeof(float)); + float* wpe_enc = (float*)malloc(T * C * sizeof(float)); + + for (int i = 0; i < B * T; ++i) { + enc_inp_cpu[i] = (int)rand() / RAND_MAX; + } + for (int i = 0; i < C * C; ++i) { + wte_enc[i] = (float)rand() / RAND_MAX; + } + for (int i = 0; i < T * C; ++i) { + wpe_enc[i] = (float)rand() / RAND_MAX; + } + + // copy data to GPU + int* enc_inp_gpu_d; + float* wte_enc_d; + float* wpe_enc_d; + cudaMalloc(&enc_inp_gpu_d, B * T * sizeof(int)); + cudaMalloc(&wte_enc_d, C * C * sizeof(float)); + cudaMalloc(&wpe_enc_d, T * C * sizeof(float)); + cudaMemcpy(enc_inp_gpu_d, enc_inp_cpu, B * T * sizeof(int), cudaMemcpyHostToDevice); + cudaMemcpy(wte_enc_d, wte_enc, C * C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(wpe_enc_d, wpe_enc, T * C * sizeof(float), cudaMemcpyHostToDevice); + + + // Allocate memory for output + float* enc_out_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* enc_out_gpu = (float*)malloc(B * T * C * sizeof(float)); + float* enc_out_gpu_d; + cudaMalloc(&enc_out_gpu_d, B * T * C * sizeof(float)); + + // Call CPU function for reference output + encoder_forward_cpu(enc_out_cpu, enc_inp_cpu, wte_enc, wpe_enc, B, T, C); + + // TODO: Call your GPU function + // encoder_forward(enc_out_gpu_d, enc_inp_gpu_d, wte_enc_d, wpe_enc_d, B, T, C); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int enc_totElements = B * T * C; + int enc_error = 0; + for (int i = 0; i < enc_totElements; i++) { + if (fabs(enc_out_cpu[i] - enc_out_gpu[i]) / (enc_out_cpu[i] + ep) > error_threshold) { + enc_error++; + } + } + + if (enc_error > 0) { + printf("+++++++++++++++ encoder_forward test failed with %d differing values +++++++++++++++\n", enc_error); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + } else { + printf("--------------------encoder_forward test passed!--------------------\n"); + } + + /*------------------------ layernorm_forward test ------------------------*/ + + // Allocate memory for input data + float* lay_inp_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* weight_ln = (float*)malloc(C * sizeof(float)); + float* bias_ln = (float*)malloc(C * sizeof(float)); + float* rstd_ln = (float*)malloc(B * T * sizeof(float)); + float* mean_ln = (float*)malloc(B * T * sizeof(float)); + + // Generate random input data + for (int i = 0; i < B * T * C; ++i) { + lay_inp_cpu[i] = (float)rand() / RAND_MAX; + } + for (int i = 0; i < B * T; ++i) { + rstd_ln[i] = (float)rand() / RAND_MAX; + mean_ln[i] = (float)rand() / RAND_MAX; + } + for (int i = 0; i < C; ++i) { + weight_ln[i] = (float)rand() / RAND_MAX; + bias_ln[i] = (float)rand() / RAND_MAX; + } + + // copy data to GPU + float* lay_inp_gpu_d; + float* weight_ln_d; + float* bias_ln_d; + float* rstd_ln_d; + float* mean_ln_d; + cudaMalloc(&lay_inp_gpu_d, B * T * C * sizeof(float)); + cudaMalloc(&weight_ln_d, C * sizeof(float)); + cudaMalloc(&bias_ln_d, C * sizeof(float)); + cudaMalloc(&rstd_ln_d, B * T * sizeof(float)); + cudaMalloc(&mean_ln_d, B * T * sizeof(float)); + cudaMemcpy(lay_inp_gpu_d, lay_inp_cpu, B * T * C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(weight_ln_d, weight_ln, C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(bias_ln_d, bias_ln, C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(rstd_ln_d, rstd_ln, B * T * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(mean_ln_d, mean_ln, B * T * sizeof(float), cudaMemcpyHostToDevice); + + // Allocate memory for output + float* lay_out_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* lay_out_gpu = (float*)malloc(B * T * C * sizeof(float)); + float* lay_out_gpu_d; + cudaMalloc(&lay_out_gpu_d, B * T * C * sizeof(float)); + + // Call CPU function for reference output + layernorm_forward_cpu(lay_out_cpu, mean_ln, rstd_ln, lay_inp_cpu, weight_ln, bias_ln, B, T, C); + + // TODO: Call your GPU function + // layernorm_forward(lay_out_gpu_d, mean_ln_d, rstd_ln_d, lay_inp_gpu_d, weight_ln_d, bias_ln_d, B, T, C); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int lay_totElements = B * T * C; + int lay_error = 0; + for (int i = 0; i < lay_totElements; i++) { + if (fabs(lay_out_cpu[i] - lay_out_gpu[i])/(lay_out_cpu[i] + ep) > error_threshold) { + lay_error++; + } + } + + if (lay_error > 0) { + printf("+++++++++++++++ layernorm_forward test failed with %d differing values +++++++++++++++\n", lay_error); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + } else { + printf("--------------------layernorm_forward test passed!--------------------\n"); + } + + /*------------------------ attention_forward test ------------------------*/ + + // Allocate memory for input data + int inp_size = B * T * max(3 * C, NH * T); + float* att_inp_cpu = (float*)malloc(inp_size * sizeof(float)); + float* qkvr = (float*)malloc(3 * B * T * C * sizeof(float)); + float* att = (float*)malloc(B * NH * NH * T * T * sizeof(float)); + // create separate copy of input for GPU to avoid input memory being overwritten + float* att_inp_gpu = (float*)malloc(inp_size * sizeof(float)); + + // Generate random input data + for (int i = 0; i < inp_size; ++i) { + att_inp_cpu[i] = (float)rand() / RAND_MAX; + att_inp_gpu[i] = att_inp_cpu[i]; + } + + // copy data to GPU + float* att_inp_gpu_d; + float* qkvr_d; + float* att_d; + cudaMalloc(&att_inp_gpu_d, inp_size * sizeof(float)); + cudaMalloc(&qkvr_d, 3 * B * T * C * sizeof(float)); + cudaMalloc(&att_d, B * NH * NH * T * T * sizeof(float)); + cudaMemcpy(att_inp_gpu_d, att_inp_gpu, inp_size * sizeof(float), cudaMemcpyHostToDevice); + + // Allocate memory for output + float* att_out_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* att_out_gpu = (float*)malloc(B * T * C * sizeof(float)); + float* att_out_gpu_d; + cudaMalloc(&att_out_gpu_d, B * T * C * sizeof(float)); + + // Call CPU function for reference output + attention_forward_cpu(att_out_cpu, qkvr, att, att_inp_cpu, B, T, C, NH); + + // TODO: Call your GPU function + // attention_forward(att_out_gpu_d, qkvr_d, att_d, att_inp_gpu_d, B, T, C, NH); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int att_totElements = B * T * C; + int att_diff = 0; + // Loop through the results to count differences + for (int i = 0; i < att_totElements; ++i) { + // Check if the result differs by more than the tolerance + if (fabs(att_out_cpu[i] - att_out_gpu[i])/(att_out_cpu[i] + ep) > error_threshold) { + ++att_diff; + } + } + + if (att_diff > 0) { + printf("+++++++++++++++ attention_forward test failed with %d differing values +++++++++++++++\n", att_diff); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + printf("++++++++ Try testing each attention component separately for easier debugging ++++++++\n"); + } else { + printf("--------------------attention_forward test passed!--------------------\n"); + } + + /*------------------------ residual_forward test ------------------------*/ + + // Allocate memory for input data + float* res_inp1_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* res_inp2_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* res_out_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* res_out_gpu = (float*)malloc(B * T * C * sizeof(float)); + + // Generate random input data + for (int i = 0; i < B * T * C; ++i) { + res_inp1_cpu[i] = (float)rand() / RAND_MAX; + res_inp2_cpu[i] = (float)rand() / RAND_MAX; + } + + // copy data to GPU + float* res_inp1_gpu_d; + float* res_inp2_gpu_d; + float* res_out_gpu_d; + cudaMalloc(&res_inp1_gpu_d, B * T * C * sizeof(float)); + cudaMalloc(&res_inp2_gpu_d, B * T * C * sizeof(float)); + cudaMalloc(&res_out_gpu_d, B * T * C * sizeof(float)); + cudaMemcpy(res_inp1_gpu_d, res_inp1_cpu, B * T * C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(res_inp2_gpu_d, res_inp2_cpu, B * T * C * sizeof(float), cudaMemcpyHostToDevice); + + // Call CPU function for reference output + residual_forward_cpu(res_out_cpu, res_inp1_cpu, res_inp2_cpu, B * T * C); + + // TODO: Call your GPU function + // residual_forward(res_out_gpu_d, res_inp1_gpu_d, res_inp2_gpu_d, B * T * C); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int res_totElements = B * T * C; + int res_diff = 0; + // Loop through the results to count differences + for (int i = 0; i < res_totElements; ++i) { + // Check if the result differs by more than the tolerance + if (fabs(res_out_cpu[i] - res_out_gpu[i])/ (res_out_cpu[i] + ep) > error_threshold) { + ++res_diff; + } + } + + if (res_diff > 0) { + printf("+++++++++++++++ residual_forward test failed with %d differing values +++++++++++++++\n", res_diff); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + } else { + printf("--------------------residual_forward test passed!--------------------\n"); + } + + /*------------------------ gelu_forward test ------------------------*/ + + // Allocate memory for input data + float* gelu_inp_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* gelu_out_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* gelu_out_gpu = (float*)malloc(B * T * C * sizeof(float)); + + // Generate random input data + for (int i = 0; i < B * T * C; ++i) { + gelu_inp_cpu[i] = (float)rand() / RAND_MAX; + } + + // copy data to GPU + float* gelu_inp_gpu_d; + float* gelu_out_gpu_d; + cudaMalloc(&gelu_inp_gpu_d, B * T * C * sizeof(float)); + cudaMalloc(&gelu_out_gpu_d, B * T * C * sizeof(float)); + cudaMemcpy(gelu_inp_gpu_d, gelu_inp_cpu, B * T * C * sizeof(float), cudaMemcpyHostToDevice); + + // Call CPU function for reference output + gelu_forward_cpu(gelu_out_cpu, gelu_inp_cpu, B * T * C); + + // TODO: Call your GPU function + // gelu_forward(gelu_out_gpu_d, gelu_inp_gpu_d, B * T * C); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int gelu_totElements = B * T * C; + int gelu_diff = 0; + for (int i = 0; i < gelu_totElements; ++i) { + if (fabs(gelu_out_cpu[i] - gelu_out_gpu[i])/(gelu_out_cpu[i] + ep) > error_threshold) { + ++gelu_diff; + } + } + + if (gelu_diff > 0) { + printf("+++++++++++++++ gelu_forward test failed with %d differing values +++++++++++++++\n", gelu_diff); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + } else { + printf("--------------------gelu_forward test passed!--------------------\n"); + } + + /*------------------------ matmul_forward test ------------------------*/ + + // Allocate memory for input data + float* mat_inp_cpu = (float*)malloc(B * T * C * sizeof(float)); + float* matmul_weight = (float*)malloc(4 * C * C * sizeof(float)); + float* matmul_bias = (float*)malloc(4 * C * sizeof(float)); + float* mat_out_cpu = (float*)malloc(B * T * 4 * C * sizeof(float)); + + // Generate random input data + for (int i = 0; i < B * T * C; ++i) { + mat_inp_cpu[i] = (float)rand() / RAND_MAX; + } + for (int i = 0; i < 4 * C * C; ++i) { + matmul_weight[i] = (float)rand() / RAND_MAX; + } + for (int i = 0; i < 4 * C; ++i) { + matmul_bias[i] = (float)rand() / RAND_MAX; + } + + // copy data to GPU + float* mat_inp_gpu_d; + float* matmul_weight_d; + float* matmul_bias_d; + float* mat_out_gpu_d; + float* mat_out_gpu = (float*)malloc(B * T * 4 * C * sizeof(float)); + cudaMalloc(&mat_inp_gpu_d, B * T * C * sizeof(float)); + cudaMalloc(&matmul_weight_d, 4 * C * C * sizeof(float)); + cudaMalloc(&matmul_bias_d, 4 * C * sizeof(float)); + cudaMalloc(&mat_out_gpu_d, B * T * 4 * C * sizeof(float)); + cudaMemcpy(mat_inp_gpu_d, mat_inp_cpu, B * T * C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(matmul_weight_d, matmul_weight, 4 * C * C * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy(matmul_bias_d, matmul_bias, 4 * C * sizeof(float), cudaMemcpyHostToDevice); + + // Call CPU function for reference output + matmul_forward_cpu(mat_out_cpu, mat_inp_cpu, matmul_weight, matmul_bias, B, T, C, OC); + + // TODO: Call your GPU function + // matmul_forward(mat_out_gpu_d, mat_inp_gpu_d, matmul_weight_d, matmul_bias_d, B, T, C, OC); + + // TODO: Copy output from your GPU function back to CPU + // cudaMemcpy(..., cudaMemcpyDeviceToHost); + + int mat_totElements = B * T * 4 * C; + int mat_diff = 0; + for (int i = 0; i < mat_totElements; ++i) { + if (fabs(mat_out_cpu[i] - mat_out_gpu[i])/(mat_out_cpu[i] + ep) > error_threshold) { + ++mat_diff; + } + } + + if (mat_diff > 0) { + printf("+++++++++++++++ matmul_forward test failed with %d differing values +++++++++++++++\n", mat_diff); + printf("+++++++++++++++ Check your implementation or error_threshold value +++++++++++++++\n"); + } else { + printf("--------------------matmul_forward test passed!--------------------\n"); + } + + printf("********************* All Random Tests Finished! *********************\n"); + + // Free allocated memory + + free(enc_inp_cpu); + free(enc_out_cpu); + free(wte_enc); + free(wpe_enc); + free(enc_out_gpu); + cudaFree(enc_inp_gpu_d); + cudaFree(wte_enc_d); + cudaFree(wpe_enc_d); + cudaFree(enc_out_gpu_d); + + free(lay_inp_cpu); + free(lay_out_cpu); + free(weight_ln); + free(bias_ln); + free(rstd_ln); + free(mean_ln); + free(lay_out_gpu); + cudaFree(lay_inp_gpu_d); + cudaFree(weight_ln_d); + cudaFree(bias_ln_d); + cudaFree(rstd_ln_d); + cudaFree(mean_ln_d); + cudaFree(lay_out_gpu_d); + + free(perm_inp_cpu); + free(perm_q_cpu); + free(perm_k_cpu); + free(perm_v_cpu); + free(perm_q_gpu); + free(perm_k_gpu); + free(perm_v_gpu); + cudaFree(perm_inp_gpu_d); + cudaFree(perm_q_d); + cudaFree(perm_k_d); + cudaFree(perm_v_d); + + free(unperm_inp_cpu); + free(unperm_out_cpu); + free(unperm_out_gpu); + cudaFree(unperm_inp_gpu_d); + cudaFree(unperm_out_gpu_d); + + free(softmax_inp_cpu); + free(softmax_inp_gpu); + free(softmax_out_cpu); + free(softmax_out_gpu); + cudaFree(softmax_inp_gpu_d); + cudaFree(softmax_out_gpu_d); + + free(att_inp_cpu); + free(att_out_cpu); + free(att_inp_gpu); + free(att_out_gpu); + free(qkvr); + free(att); + cudaFree(att_inp_gpu_d); + cudaFree(qkvr_d); + cudaFree(att_d); + cudaFree(att_out_gpu_d); + + free(res_inp1_cpu); + free(res_inp2_cpu); + free(res_out_cpu); + free(res_out_gpu); + cudaFree(res_inp1_gpu_d); + cudaFree(res_inp2_gpu_d); + cudaFree(res_out_gpu_d); + + free(gelu_inp_cpu); + free(gelu_out_cpu); + free(gelu_out_gpu); + cudaFree(gelu_inp_gpu_d); + cudaFree(gelu_out_gpu_d); + + free(mat_inp_cpu); + free(mat_out_gpu); + free(matmul_weight); + free(matmul_bias); + cudaFree(mat_inp_gpu_d); + cudaFree(matmul_weight_d); + cudaFree(matmul_bias_d); + cudaFree(mat_out_gpu_d); + + return 0; +} + diff --git a/test.sh b/test.sh new file mode 100644 index 0000000..0fd5535 --- /dev/null +++ b/test.sh @@ -0,0 +1,137 @@ +#!/bin/bash +set -euo pipefail + +shopt -s nullglob + +# ------------------------- +# parse command line arguments +# ------------------------- +SUFFIX="" +DO_TEST=0 +DO_NSYS=0 +DO_NCU=0 + +while [[ $# -gt 0 ]]; do + case $1 in + --test) + DO_TEST=1 + shift + ;; + --nsys) + DO_NSYS=1 + shift + ;; + --ncu) + DO_NCU=1 + shift + ;; + *) + if [[ -z "$SUFFIX" ]]; then + SUFFIX="$1" + else + echo "Error: multiple non-option arguments provided ('$SUFFIX' and '$1')" + exit 1 + fi + shift + ;; + esac +done + +if [[ -z "$SUFFIX" ]]; then + echo "Usage: $0 [--test] [--nsys] [--ncu]" + exit 1 +fi + +echo "Suffix: $SUFFIX" + +# ------------------------- +# kernel swap (temporary replacement) +# ------------------------- +SRC_DIR="kernels_${SUFFIX}" +KERNEL_DIR="kernels" + +if [[ ! -d "$SRC_DIR" ]]; then + echo "Error: source directory '$SRC_DIR' does not exist." + exit 1 +fi + +src_files=("$SRC_DIR"/*.cuh) +if [[ ${#src_files[@]} -eq 0 ]]; then + echo "Error: no .cuh files found in '$SRC_DIR'" + exit 1 +fi + +BACKUP_DIR=".kernel_backup_${SUFFIX}_$(date +%s)" +mkdir -p "$BACKUP_DIR" + +backup_files=("$KERNEL_DIR"/*.cuh) +if [[ ${#backup_files[@]} -gt 0 ]]; then + echo "Backing up current kernel files to $BACKUP_DIR" + cp "${backup_files[@]}" "$BACKUP_DIR/" +else + echo "No existing .cuh files in $KERNEL_DIR to back up" +fi + +# ------------------------- +# trap: always restore on exit (success, error, or signal) +# ------------------------- +trap ' + echo "Restoring original kernel files from backup..." + restore_files=("$BACKUP_DIR"/*.cuh) + if [[ ${#restore_files[@]} -gt 0 ]]; then + for f in "${restore_files[@]}"; do + cp "$f" "$KERNEL_DIR/" + done + echo "Restoration complete." + else + echo "Warning: no backup files found in $BACKUP_DIR — kernel directory may contain swapped files." + fi + rm -rf "$BACKUP_DIR" +' EXIT + +echo "Swapping kernel files from $SRC_DIR to $KERNEL_DIR" +for f in "${src_files[@]}"; do + base=$(basename "$f") + cp "$f" "$KERNEL_DIR/$base" +done + +# ------------------------- +# build the executable +# ------------------------- +echo "Building test_gpt2_${SUFFIX}..." +make test_gpt2 "$SUFFIX" # if this fails, trap fires and restores + +BIN="./build/test_gpt2_${SUFFIX}" + +# ========================================================= +# TEST RUN (correctness) +# ========================================================= +if [[ $DO_TEST -eq 1 ]]; then + echo "Running TEST (sbatch verify.slurm $SUFFIX)..." + sbatch verify.slurm "$SUFFIX" +fi + +# ========================================================= +# NSYS RUN (profiling with Nsight Systems) +# ========================================================= +if [[ $DO_NSYS -eq 1 ]]; then + echo "Running NSYS (sbatch verify_nsys.slurm $SUFFIX)..." + sbatch verify_nsys.slurm "$SUFFIX" +fi + +# ========================================================= +# NCU RUN (profiling with Nsight Compute) +# ========================================================= +if [[ $DO_NCU -eq 1 ]]; then + echo "Running NCU (sbatch verify_ncu.slurm $SUFFIX)..." + sbatch verify_ncu.slurm "$SUFFIX" +fi + +# ------------------------- +# final status +# ------------------------- +echo "Checking Slurm queue for user $USER..." +squeue -u "$USER" + +echo "Done." +# trap fires here on clean exit too — restore always runs \ No newline at end of file diff --git a/test_gpt2.cu b/test_gpt2.cu new file mode 100644 index 0000000..f180b19 --- /dev/null +++ b/test_gpt2.cu @@ -0,0 +1,138 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include "utils/utils.h" +#include "gpt2.cuh" + +int main(int argc, char *argv[]) { + + // set up the device + int deviceIdx = 0; + cudaCheck(cudaSetDevice(deviceIdx)); + cudaDeviceProp deviceProp; + cudaGetDeviceProperties(&deviceProp, deviceIdx); + + // setup cuBLAS and cuBLASLt + cublasCheck(cublasCreate(&cublas_handle)); + int enable_tf32 = 0; + // enable_tf32 = deviceProp.major >= 8 ? 1 : 0; + printf("test-gpt2-kernels - TF32 Mode is turned %s\n", enable_tf32 ? "ON." : "OFF."); + cublas_compute_type = enable_tf32 ? CUBLAS_COMPUTE_32F_FAST_TF32 : CUBLAS_COMPUTE_32F; + cublasMath_t cublas_math_mode = enable_tf32 ? CUBLAS_TF32_TENSOR_OP_MATH : CUBLAS_DEFAULT_MATH; + cublasCheck(cublasSetMathMode(cublas_handle, cublas_math_mode)); + + // build the GPT-2 model from a checkpoint + GPT2 model; + gpt2_build_from_checkpoint(&model, "/work/hdd/bche/Project_GPT/gpt2_124M.bin"); + + // int C = model.config.channels; + int V = model.config.vocab_size; + int Vp = model.config.padded_vocab_size; + int maxT = model.config.max_seq_len; + // int L = model.config.num_layers; + + // load additional information that we will use for debugging and error checking + FILE *state_file = fopenCheck("/work/hdd/bche/Project_GPT/gpt2_inference_verif.bin", "rb"); + int state_header[256]; + freadCheck(state_header, sizeof(int), 256, state_file); + if (state_header[0] != 20240327) { printf("test-gpt2 - Bad magic state file\n"); exit(EXIT_FAILURE); } + if (state_header[1] != 2) { + fprintf(stderr, "test-gpt2 - Bad version in state file\n"); + exit(EXIT_FAILURE); + } + int B = state_header[2]; // batch size, e.g. 4 + int T = state_header[3]; // time / sequence length (e.g. 64, up to maxT) + assert(0 <= T && T <= maxT); + printf("test-gpt2 - [State]\n"); + printf("Batch Size: %d\n", B); + printf("Sequence Length: %d\n", T); + + // inputs and expected outputs, only used for error checking + int* x = (int*)mallocCheck(B * T * sizeof(int)); + float* expected_logits = (float*) mallocCheck(B * T * V * sizeof(float)); + + // read reference information + freadCheck(x, sizeof(int), B*T, state_file); + // freadCheck(y, sizeof(int), B*T, state_file); + fseekCheck(state_file, B*T*sizeof(int), SEEK_CUR); + freadCheck(expected_logits, sizeof(float), B*T*V, state_file); + // freadCheck(expected_loss, sizeof(float), 1, state_file); + // freadCheck(expected_grads_memory, sizeof(float), model.num_parameters, state_file); + fcloseCheck(state_file); + + // overall OK signal for the test + int allok = 1; + + // First, do target-free forward pass to validate logits + gpt2_forward(&model, x, B, T); + // at this point, target should be equal to expected_logits, let's compare + // copy logits to CPU so we can compare them + float* logits_cpu = (float*)mallocCheck(B * T * Vp * sizeof(float)); + cudaCheck(cudaMemcpy(logits_cpu, model.acts.output, B * T * Vp * sizeof(float), cudaMemcpyDeviceToHost)); + + // compare the output logits from the forward pass + // also careful that we don't access and compare the padded columns of logits + int logits_ok = 1; + double maxerr = 0.0f; + double rmse = 0; + printf("test-gpt2 - Printing 10 Expected vs Actual Logits: \n"); + for (int bt = 0; bt < B*T; bt++) { + for (int v = 0; v < V; v++) { + int i = bt * Vp + v; // linearized index + float expected = expected_logits[bt*V + v]; + if (i < 10) { + printf("%f, %f\n", expected, logits_cpu[i]); + } + if (isnan(logits_cpu[i])) { + printf("=========================MISMATCH AT INDEX %d,%d: ", bt, v); + printf("%f %f (logit is nan)=========================\n", expected, logits_cpu[i]); + logits_ok = 0; + bt = B*T; // to break out of both loops + break; + } + double abserr = abs(logits_cpu[i] - expected); + double relerr = abs(logits_cpu[i] - expected) / fmaxf(1e-3, abs(expected)); + double err = fmin(abserr, relerr); + maxerr = fmaxf(maxerr, err); + rmse += err * err; + if (err > 0.1) { + printf("=========================MISMATCH AT INDEX %d,%d: ", bt, v); + printf("%f %f (err=%f)=========================\n", expected, logits_cpu[i], err); + logits_ok = 0; + bt = B*T; // to break out of both loops + break; + } + } + } + rmse = sqrt(rmse / (B*T*V)); + if (rmse > 0.004) { + printf("================ RMSE is too high! (max err %f, RMSE %f) ==================\n", maxerr, rmse); + logits_ok = 0; + } + allok = allok && logits_ok; + if(!logits_ok) { + printf("================ Logits NOT OK! ================\n"); + } + else { + printf("---------------- Logits OK! (max err %f, RMSE %f) ----------------\n", maxerr, rmse); + } + + // final approval + if (allok) { + printf("_________________________ All tests passed! _________________________\n"); + } else { + printf("_________________________ Some tests failed! _________________________\n"); + } + + // free everything + free(x); + free(expected_logits); + gpt2_free(&model); + cublasCheck(cublasDestroy(cublas_handle)); + + return 0; +} \ No newline at end of file diff --git a/test_gpt2_kernels.cu b/test_gpt2_kernels.cu new file mode 100644 index 0000000..fd3c1f4 --- /dev/null +++ b/test_gpt2_kernels.cu @@ -0,0 +1,208 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#include "utils/utils.h" +#include "gpt2.cuh" + +void check_and_swap_acts(const char * kernel, FILE * kernel_state, float * output, size_t n) { + cudaError_t lasterror = cudaGetLastError(); + if (lasterror != cudaSuccess) { + printf("CUDA error running kernel %s: %s\n", kernel, cudaGetErrorString(lasterror)); + exit(EXIT_FAILURE); + } + + float * output_cpu = (float*)mallocCheck(n * sizeof(float)); + cudaCheck(cudaMemcpy(output_cpu, output, n * sizeof(float), cudaMemcpyDeviceToHost)); + float * kernel_cpu = (float*)mallocCheck(n * sizeof(float)); + freadCheck(kernel_cpu, sizeof(float), n, kernel_state); + int mismatches = 0; + double maxerr = 0; + double rmse = 0; + for (size_t i = 0; i < n; i++) { + if (isnan(output_cpu[i])) { + mismatches++; + continue; + } + double abserr = abs(output_cpu[i] - kernel_cpu[i]); + double relerr = abs(output_cpu[i] - kernel_cpu[i]) / fmaxf(1e-3, abs(kernel_cpu[i])); + double err = fmin(abserr, relerr); + maxerr = fmax(maxerr, err); + rmse += err * err; + if (err > 0.05) { // 5% and 0.05 error threshold + mismatches++; + // printf("Mismatch at index %zu: %f vs %f\n", i, output_cpu[i], kernel_cpu[i]); + } + } + rmse = sqrt(rmse/n); + if (mismatches > 0) { + printf("===================== %s KERNEL FAILED: %d mismatches (max err %f, RMSE %f) =====================\n", kernel, mismatches, maxerr, rmse); + } else if (rmse > 0.001) { + printf("===================== %s KERNEL FAILED: RMSE (max err %f, RMSE %f) =====================\n", kernel, maxerr, rmse); + } else { + printf("--------------------- %s Kernel Passed (max err %f, RMSE %f) ---------------------\n", kernel, maxerr, rmse); + } + // replace with expected output + cudaCheck(cudaMemcpy(output, kernel_cpu, n * sizeof(float), cudaMemcpyHostToDevice)); + free(output_cpu); + free(kernel_cpu); +} + +int main(int argc, char *argv[]) { + printf("Starting test_gpt2_kernels\n"); + + // set up the device + int deviceIdx = 0; + cudaCheck(cudaSetDevice(deviceIdx)); + cudaDeviceProp deviceProp; + cudaGetDeviceProperties(&deviceProp, deviceIdx); + + // setup cuBLAS and cuBLASLt + cublasCheck(cublasCreate(&cublas_handle)); + int enable_tf32 = 0; + // enable_tf32 = deviceProp.major >= 8 ? 1 : 0; + printf("test-gpt2-kernels - TF32 Mode is turned %s\n", enable_tf32 ? "ON." : "OFF."); + cublas_compute_type = enable_tf32 ? CUBLAS_COMPUTE_32F_FAST_TF32 : CUBLAS_COMPUTE_32F; + cublasMath_t cublas_math_mode = enable_tf32 ? CUBLAS_TF32_TENSOR_OP_MATH : CUBLAS_DEFAULT_MATH; + cublasCheck(cublasSetMathMode(cublas_handle, cublas_math_mode)); + + // build the GPT-2 model from a checkpoint + GPT2 modelval; + GPT2 * model = &modelval; + gpt2_build_from_checkpoint(model, "/work/hdd/bche/Project_GPT/gpt2_124M.bin"); + + int C = model->config.channels; + int V = model->config.vocab_size; + int maxT = model->config.max_seq_len; + int L = model->config.num_layers; + int NH = model->config.num_heads; + + + // load additional information that we will use for debugging and error checking + FILE *state_file = fopenCheck("/work/hdd/bche/Project_GPT/gpt2_kernel_verif.bin", "rb"); + int state_header[256]; + freadCheck(state_header, sizeof(int), 256, state_file); + if (state_header[0] != 20250211) { printf("test-gpt2-kernels - Bad magic kernel state file\n"); exit(EXIT_FAILURE); } + if (state_header[1] != 1) { + fprintf(stderr, "test-gpt2-kernels - Bad version in kernel state file\n"); + exit(EXIT_FAILURE); + } + int B = state_header[2]; // batch size, e.g. 4 + int T = state_header[3]; // time / sequence length (e.g. 64, up to maxT) + assert(0 <= T && T <= maxT); + + // read inputs + int* input_tokens = (int*)mallocCheck(B * T * sizeof(int)); + freadCheck(input_tokens, sizeof(int), B*T, state_file); + + // validate inputs, all indices must be in the range [0, V) + for(int i = 0; i < B * T; i++) { + assert(0 <= input_tokens[i] && input_tokens[i] < V); + } + + // allocate space for all the activations + model->batch_size = B; + model->seq_len = T; + fill_in_activation_sizes(model->act_sizes, B, T, model->config); + size_t num_activations = 0; + for (size_t i = 0; i < NUM_ACTIVATION_TENSORS; i++) { + num_activations += model->act_sizes[i]; + } + model->num_activations = num_activations; + model->acts_memory = malloc_and_point_activations(&model->acts, model->act_sizes); + printf("test-gpt2-kernels - Allocated %zu MiB for activations\n", (num_activations * sizeof(float)) >> 20); // >> 20 is /(1024*1024) + cudaCheck(cudaMalloc((void**)&model->inputs, B * T * sizeof(int))); + cudaCheck(cudaMallocHost((void**)&model->cpu_losses, B * T * sizeof(float))); + + // copy inputs to the model + cudaCheck(cudaMemcpy(model->inputs, input_tokens, B * T * sizeof(int), cudaMemcpyHostToDevice)); + + // forward pass + ParameterTensors params = model->params; // for brevity + ActivationTensors acts = model->acts; + float* residual; + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe, B, T, C); // encoding goes into residual[0] + check_and_swap_acts("encoder", state_file, acts.encoded, B * T * C * sizeof(float)); + for (int l = 0; l < L; l++) { + + residual = l == 0 ? acts.encoded : acts.residual3 + (l-1) * B * T * C; + + // get the pointers of the weights for this layer + float* l_ln1w = params.ln1w + l * C; + float* l_ln1b = params.ln1b + l * C; + float* l_qkvw = params.qkvw + l * 3*C * C; + float* l_qkvb = params.qkvb + l * 3*C; + float* l_attprojw = params.attprojw + l * C * C; + float* l_attprojb = params.attprojb + l * C; + float* l_ln2w = params.ln2w + l * C; + float* l_ln2b = params.ln2b + l * C; + float* l_fcw = params.fcw + l * 4*C * C; + float* l_fcb = params.fcb + l * 4*C; + float* l_fcprojw = params.fcprojw + l * C * 4*C; + float* l_fcprojb = params.fcprojb + l * C; + + // get the pointers of the activations for this layer + float* l_ln1 = acts.ln1 + l * B * T * C; + float* l_ln1_mean = acts.ln1_mean + l * B * T; + float* l_ln1_rstd = acts.ln1_rstd + l * B * T; + float* l_qkvr = acts.qkvr + l * B * T * 3*C; + float* l_atty = acts.atty + l * B * T * C; + float* l_att = acts.att + l * B * NH * T * T; + float* l_attproj = acts.attproj + l * B * T * C; + float* l_residual2 = acts.residual2 + l * B * T * C; + float* l_ln2 = acts.ln2 + l * B * T * C; + float* l_ln2_mean = acts.ln2_mean + l * B * T; + float* l_ln2_rstd = acts.ln2_rstd + l * B * T; + float* l_fch = acts.fch + l * B * T * 4*C; + float* l_fch_gelu = acts.fch_gelu + l * B * T * 4*C; + float* l_fcproj = acts.fcproj + l * B * T * C; + float* l_residual3 = acts.residual3 + l * B * T * C; + // these are only needed as scratchpads for the forward pass, but + // need not be stored for backward + float* scratch = acts.output; + + printf("\n______________________________ Layer %d ______________________________\n", l); + + // now do the forward pass + layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C); + check_and_swap_acts("layernorm", state_file, l_ln1, B * T * C); + matmul_forward(scratch, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C); + check_and_swap_acts("matmul", state_file, scratch, B * T * 3*C); + attention_forward(l_atty, l_qkvr, l_att, scratch, B, T, C, NH); + check_and_swap_acts("attention", state_file, l_atty, B * T * C); + matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C); + check_and_swap_acts("matmul", state_file, l_attproj, B * T * C); + residual_forward(l_residual2, residual, l_attproj, B*T*C); + check_and_swap_acts("residual", state_file, l_residual2, B * T * C); + layernorm_forward(l_ln2, l_ln2_mean, l_ln2_rstd, l_residual2, l_ln2w, l_ln2b, B, T, C); + check_and_swap_acts("layernorm", state_file, l_ln2, B * T * C); + matmul_forward(l_fch, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + check_and_swap_acts("matmul", state_file, l_fch, B * T * 4*C); + gelu_forward(l_fch_gelu, l_fch, B*T*4*C); + check_and_swap_acts("gelu", state_file, l_fch_gelu, B * T * 4*C); + matmul_forward(l_fcproj, l_fch_gelu, l_fcprojw, l_fcprojb, B, T, 4*C, C); + check_and_swap_acts("matmul", state_file, l_fcproj, B * T * C); + residual_forward(l_residual3, l_residual2, l_fcproj, B*T*C); + check_and_swap_acts("residual", state_file, l_residual3, B * T * C); + + } + + residual = acts.residual3 + (L-1) * B * T * C; // last residual is in residual3 + layernorm_forward(acts.lnf, acts.lnf_mean, acts.lnf_rstd, residual, params.lnfw, params.lnfb, B, T, C); + check_and_swap_acts("layernorm", state_file, acts.lnf, B * T * C); + + printf("_______________________________ All kernel tests done! _______________________________\n"); + + // close the file + fcloseCheck(state_file); + + // free everything + free(input_tokens); + gpt2_free(model); + cublasCheck(cublasDestroy(cublas_handle)); + + return 0; +} \ No newline at end of file diff --git a/test_gpt2_nsys_prof_base.out b/test_gpt2_nsys_prof_base.out new file mode 100644 index 0000000..8c97b63 --- /dev/null +++ b/test_gpt2_nsys_prof_base.out @@ -0,0 +1,106 @@ +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-a416.qdstrm' + [1/8] [0% ] report1.nsys-rep [1/8] [0% ] report1.nsys-rep [1/8] [====25% ] report1.nsys-rep [1/8] [==========47% ] report1.nsys-rep [1/8] [==========48% ] report1.nsys-rep [1/8] [==========49% ] report1.nsys-rep [1/8] [===========50% ] report1.nsys-rep [1/8] [======34% ] report1.nsys-rep [1/8] [======35% ] report1.nsys-rep [1/8] [===========50% ] report1.nsys-rep [1/8] [===========51% ] report1.nsys-rep [1/8] [===========52% ] report1.nsys-rep [1/8] [===========53% ] report1.nsys-rep [1/8] [==================77% ] report1.nsys-rep [1/8] [======================91% ] report1.nsys-rep [1/8] [========================100%] report1.nsys-rep [1/8] [========================100%] report1.nsys-rep + [2/8] [0% ] report1.sqlite [2/8] [1% ] report1.sqlite [2/8] [2% ] report1.sqlite [2/8] [3% ] report1.sqlite [2/8] [4% ] report1.sqlite [2/8] [5% ] report1.sqlite [2/8] [6% ] report1.sqlite [2/8] [7% ] report1.sqlite [2/8] [8% ] report1.sqlite [2/8] [9% ] report1.sqlite [2/8] [10% ] report1.sqlite [2/8] [11% ] report1.sqlite [2/8] [12% ] report1.sqlite [2/8] [13% ] report1.sqlite [2/8] [14% ] report1.sqlite [2/8] [=15% ] report1.sqlite [2/8] [=16% ] report1.sqlite [2/8] [=17% ] report1.sqlite [2/8] [==18% ] report1.sqlite [2/8] [==19% ] report1.sqlite [2/8] [==20% ] report1.sqlite [2/8] [==21% ] report1.sqlite [2/8] [===22% ] report1.sqlite [2/8] [===23% ] report1.sqlite [2/8] [===24% ] report1.sqlite [2/8] [====25% ] report1.sqlite [2/8] [====26% ] report1.sqlite [2/8] [====27% ] report1.sqlite [2/8] [====28% ] report1.sqlite [2/8] [=====29% ] report1.sqlite [2/8] [=====30% ] report1.sqlite [2/8] [=====31% ] report1.sqlite [2/8] [=====32% ] report1.sqlite [2/8] [======33% ] report1.sqlite [2/8] [======34% ] report1.sqlite [2/8] [======35% ] report1.sqlite [2/8] [=======36% ] report1.sqlite [2/8] [=======37% ] report1.sqlite [2/8] [=======38% ] report1.sqlite [2/8] [=======39% ] report1.sqlite [2/8] [========40% ] report1.sqlite [2/8] [========41% ] report1.sqlite [2/8] [========42% ] report1.sqlite [2/8] [=========43% ] report1.sqlite [2/8] [=========44% ] report1.sqlite [2/8] [=========45% ] report1.sqlite [2/8] [=========46% ] report1.sqlite [2/8] [==========47% ] report1.sqlite [2/8] [==========48% ] report1.sqlite [2/8] [==========49% ] report1.sqlite [2/8] [===========50% ] report1.sqlite [2/8] [===========51% ] report1.sqlite [2/8] [===========52% ] report1.sqlite [2/8] [===========53% ] report1.sqlite [2/8] [============54% ] report1.sqlite [2/8] [============55% ] report1.sqlite [2/8] [============56% ] report1.sqlite [2/8] [============57% ] report1.sqlite [2/8] [=============58% ] report1.sqlite [2/8] [=============59% ] report1.sqlite [2/8] [=============60% ] report1.sqlite [2/8] [==============61% ] report1.sqlite [2/8] [==============62% ] report1.sqlite [2/8] [==============63% ] report1.sqlite [2/8] [==============64% ] report1.sqlite [2/8] [===============65% ] report1.sqlite [2/8] [===============66% ] report1.sqlite [2/8] [===============67% ] report1.sqlite [2/8] [================68% ] report1.sqlite [2/8] [================69% ] report1.sqlite [2/8] [================70% ] report1.sqlite [2/8] [================71% ] report1.sqlite [2/8] [=================72% ] report1.sqlite [2/8] [=================73% ] report1.sqlite [2/8] [=================74% ] report1.sqlite [2/8] [==================75% ] report1.sqlite [2/8] [==================76% ] report1.sqlite [2/8] [==================77% ] report1.sqlite [2/8] [==================78% ] report1.sqlite [2/8] [===================79% ] report1.sqlite [2/8] [===================80% ] report1.sqlite [2/8] [===================81% ] report1.sqlite [2/8] [===================82% ] report1.sqlite [2/8] [====================83% ] report1.sqlite [2/8] [====================84% ] report1.sqlite [2/8] [====================85% ] report1.sqlite [2/8] [=====================86% ] report1.sqlite [2/8] [=====================87% ] report1.sqlite [2/8] [=====================88% ] report1.sqlite [2/8] [=====================89% ] report1.sqlite [2/8] [======================90% ] report1.sqlite [2/8] [======================91% ] report1.sqlite [2/8] [======================92% ] report1.sqlite [2/8] [=======================93% ] report1.sqlite [2/8] [=======================94% ] report1.sqlite [2/8] [=======================95% ] report1.sqlite [2/8] [=======================96% ] report1.sqlite [2/8] [========================97% ] report1.sqlite [2/8] [========================98% ] report1.sqlite [2/8] [========================99% ] report1.sqlite [2/8] [========================100%] report1.sqlite [2/8] [========================100%] report1.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 72.0 705681437 17 41510672.8 22601905.0 1263 105099462 46154890.8 poll + 12.0 117682190 590 199461.3 23083.0 1022 21896375 998390.5 ioctl + 8.9 86890629 6 14481771.5 81337.5 7083 79381817 31927343.1 fread + 4.5 44498140 38 1171003.7 6392.0 3587 43521912 7056305.1 fopen + 1.3 12418008 1 12418008.0 12418008.0 12418008 12418008 0.0 pthread_cond_wait + 0.7 7195204 1 7195204.0 7195204.0 7195204 7195204 0.0 pthread_cond_broadcast + 0.2 2355779 25 94231.2 8807.0 6823 1637274 324132.7 mmap64 + 0.1 1272540 10 127254.0 31053.0 21640 545090 194577.0 write + 0.1 726199 32 22693.7 2144.0 1343 361726 80378.1 fclose + 0.1 601015 17 35353.8 7755.0 1883 391553 94165.5 mmap + 0.0 368309 2 184154.5 184154.5 168145 200164 22640.9 pthread_create + 0.0 316921 55 5762.2 5831.0 2354 10750 2150.6 open64 + 0.0 78106 1 78106.0 78106.0 78106 78106 0.0 fgets + 0.0 63408 1 63408.0 63408.0 63408 63408 0.0 fflush + 0.0 48070 9 5341.1 4538.0 2655 10159 2577.4 munmap + 0.0 47368 6 7894.7 6457.0 3056 16481 4729.8 open + 0.0 21011 3 7003.7 8336.0 3808 8867 2780.2 pipe2 + 0.0 18394 10 1839.4 1417.5 1021 3646 945.4 read + 0.0 16351 2 8175.5 8175.5 6753 9598 2011.7 socket + 0.0 15940 1 15940.0 15940.0 15940 15940 0.0 connect + 0.0 6122 1 6122.0 6122.0 6122 6122 0.0 fopen64 + 0.0 3316 2 1658.0 1658.0 1543 1773 162.6 fwrite + 0.0 2034 1 2034.0 2034.0 2034 2034 0.0 bind + 0.0 1142 1 1142.0 1142.0 1142 1142 0.0 fcntl + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- --------------------------------- + 95.9 293359199 3 97786399.7 36170906.0 18784 257169509 139208389.2 cudaMemcpy + 1.7 5152663 12 429388.6 6382.0 190 1823102 617871.7 cudaFree + 0.8 2489648 1 2489648.0 2489648.0 2489648 2489648 0.0 cudaFreeHost + 0.5 1393628 1 1393628.0 1393628.0 1393628 1393628 0.0 cudaGetDeviceProperties_v2_v12000 + 0.5 1378710 1 1378710.0 1378710.0 1378710 1378710 0.0 cudaMallocHost + 0.3 970320 171 5674.4 2835.0 2715 346198 26465.2 cudaLaunchKernel + 0.3 874376 6 145729.3 126041.0 3197 338894 137484.8 cudaMalloc + 0.1 155420 838 185.5 150.0 80 4749 179.1 cuGetProcAddress_v2 + 0.0 33162 1 33162.0 33162.0 33162 33162 0.0 cudaMemset + 0.0 19137 18 1063.2 301.0 250 8406 1995.0 cudaEventCreateWithFlags + 0.0 13998 4 3499.5 2850.5 1113 7184 2591.7 cudaDeviceSynchronize + 0.0 8342 18 463.4 296.0 210 2375 525.8 cudaEventDestroy + 0.0 5170 4 1292.5 1307.5 1142 1413 132.0 cuInit + 0.0 1463 3 487.7 180.0 110 1173 594.5 cuModuleGetLoadingMode + 0.0 832 2 416.0 416.0 261 571 219.2 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- --------- --------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 97.4 243410880 49 4967569.0 4540231.0 1227010 73760510 10134533.5 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 1.6 4088678 25 163547.1 163712.0 158528 166049 1938.8 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 0.5 1133283 12 94440.3 94448.5 94144 94656 137.9 preattention_scores_kernel(float *, float *, int, int, int, int, float *) + 0.1 345409 12 28784.1 28448.0 28192 29889 625.8 attention_v_kernel(float *, float *, float *, int, int, int, int) + 0.1 227136 12 18928.0 18944.0 18784 19008 57.1 gelu_forward_kernel(float *, const float *, int) + 0.1 215745 12 17978.8 17952.0 17824 18208 112.4 softmax_forward_kernel(float *, float, const float *, int, int) + 0.1 145632 24 6068.0 6080.0 5632 6592 393.7 residual_forward_kernel(float *, float *, float *, int) + 0.0 117472 12 9789.3 9760.0 9632 10080 114.5 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.0 86400 1 86400.0 86400.0 86400 86400 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.0 70560 12 5880.0 5888.0 5824 5888 19.9 unpermute_kernel(float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 81.6 36080190 2 18040095.0 18040095.0 864 36079326 25511325.1 [CUDA memcpy Host-to-Device] + 17.5 7741294 1 7741294.0 7741294.0 7741294 7741294 0.0 [CUDA memcpy Device-to-Host] + 0.9 408288 1 408288.0 408288.0 408288 408288 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/report1.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/report1.sqlite diff --git a/test_gpt2_nsys_prof_req_0.out b/test_gpt2_nsys_prof_req_0.out new file mode 100644 index 0000000..6eb8e25 --- /dev/null +++ b/test_gpt2_nsys_prof_req_0.out @@ -0,0 +1,107 @@ +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431740 +-39.836346, -39.836452 +-43.065910, -43.066021 +-42.828045, -42.828144 +-43.529541, -43.529644 +-44.318398, -44.318497 +-41.227425, -41.227528 +-41.270760, -41.270863 +-42.541393, -42.541523 +-42.394997, -42.395119 +---------------- Logits OK! (max err 0.000043, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-e0c8.qdstrm' + [1/8] [0% ] report4.nsys-rep [1/8] [0% ] report4.nsys-rep [1/8] [====25% ] report4.nsys-rep [1/8] [==========49% ] report4.nsys-rep [1/8] [===========50% ] report4.nsys-rep [1/8] [=======36% ] report4.nsys-rep [1/8] [===========50% ] report4.nsys-rep [1/8] [===========51% ] report4.nsys-rep [1/8] [===========52% ] report4.nsys-rep [1/8] [===========53% ] report4.nsys-rep [1/8] [============54% ] report4.nsys-rep [1/8] [============55% ] report4.nsys-rep [1/8] [============56% ] report4.nsys-rep [1/8] [============57% ] report4.nsys-rep [1/8] [=============58% ] report4.nsys-rep [1/8] [=============59% ] report4.nsys-rep [1/8] [=============60% ] report4.nsys-rep [1/8] [===================79% ] report4.nsys-rep [1/8] [======================90% ] report4.nsys-rep [1/8] [========================100%] report4.nsys-rep [1/8] [========================100%] report4.nsys-rep + [2/8] [0% ] report4.sqlite [2/8] [1% ] report4.sqlite [2/8] [2% ] report4.sqlite [2/8] [3% ] report4.sqlite [2/8] [4% ] report4.sqlite [2/8] [5% ] report4.sqlite [2/8] [6% ] report4.sqlite [2/8] [7% ] report4.sqlite [2/8] [8% ] report4.sqlite [2/8] [9% ] report4.sqlite [2/8] [10% ] report4.sqlite [2/8] [11% ] report4.sqlite [2/8] [12% ] report4.sqlite [2/8] [13% ] report4.sqlite [2/8] [14% ] report4.sqlite [2/8] [=15% ] report4.sqlite [2/8] [=16% ] report4.sqlite [2/8] [=17% ] report4.sqlite [2/8] [==18% ] report4.sqlite [2/8] [==19% ] report4.sqlite [2/8] [==20% ] report4.sqlite [2/8] [==21% ] report4.sqlite [2/8] [===22% ] report4.sqlite [2/8] [===23% ] report4.sqlite [2/8] [===24% ] report4.sqlite [2/8] [====25% ] report4.sqlite [2/8] [====26% ] report4.sqlite [2/8] [====27% ] report4.sqlite [2/8] [====28% ] report4.sqlite [2/8] [=====29% ] report4.sqlite [2/8] [=====30% ] report4.sqlite [2/8] [=====31% ] report4.sqlite [2/8] [=====32% ] report4.sqlite [2/8] [======33% ] report4.sqlite [2/8] [======34% ] report4.sqlite [2/8] [======35% ] report4.sqlite [2/8] [=======36% ] report4.sqlite [2/8] [=======37% ] report4.sqlite [2/8] [=======38% ] report4.sqlite [2/8] [=======39% ] report4.sqlite [2/8] [========40% ] report4.sqlite [2/8] [========41% ] report4.sqlite [2/8] [========42% ] report4.sqlite [2/8] [=========43% ] report4.sqlite [2/8] [=========44% ] report4.sqlite [2/8] [=========45% ] report4.sqlite [2/8] [=========46% ] report4.sqlite [2/8] [==========47% ] report4.sqlite [2/8] [==========48% ] report4.sqlite [2/8] [==========49% ] report4.sqlite [2/8] [===========50% ] report4.sqlite [2/8] [===========51% ] report4.sqlite [2/8] [===========52% ] report4.sqlite [2/8] [===========53% ] report4.sqlite [2/8] [============54% ] report4.sqlite [2/8] [============55% ] report4.sqlite [2/8] [============56% ] report4.sqlite [2/8] [============57% ] report4.sqlite [2/8] [=============58% ] report4.sqlite [2/8] [=============59% ] report4.sqlite [2/8] [=============60% ] report4.sqlite [2/8] [==============61% ] report4.sqlite [2/8] [==============62% ] report4.sqlite [2/8] [==============63% ] report4.sqlite [2/8] [==============64% ] report4.sqlite [2/8] [===============65% ] report4.sqlite [2/8] [===============66% ] report4.sqlite [2/8] [===============67% ] report4.sqlite [2/8] [================68% ] report4.sqlite [2/8] [================69% ] report4.sqlite [2/8] [================70% ] report4.sqlite [2/8] [================71% ] report4.sqlite [2/8] [=================72% ] report4.sqlite [2/8] [=================73% ] report4.sqlite [2/8] [=================74% ] report4.sqlite [2/8] [==================75% ] report4.sqlite [2/8] [==================76% ] report4.sqlite [2/8] [==================77% ] report4.sqlite [2/8] [==================78% ] report4.sqlite [2/8] [===================79% ] report4.sqlite [2/8] [===================80% ] report4.sqlite [2/8] [===================81% ] report4.sqlite [2/8] [===================82% ] report4.sqlite [2/8] [====================83% ] report4.sqlite [2/8] [====================84% ] report4.sqlite [2/8] [====================85% ] report4.sqlite [2/8] [=====================86% ] report4.sqlite [2/8] [=====================87% ] report4.sqlite [2/8] [=====================88% ] report4.sqlite [2/8] [=====================89% ] report4.sqlite [2/8] [======================90% ] report4.sqlite [2/8] [======================91% ] report4.sqlite [2/8] [======================92% ] report4.sqlite [2/8] [=======================93% ] report4.sqlite [2/8] [=======================94% ] report4.sqlite [2/8] [=======================95% ] report4.sqlite [2/8] [=======================96% ] report4.sqlite [2/8] [========================97% ] report4.sqlite [2/8] [========================98% ] report4.sqlite [2/8] [========================99% ] report4.sqlite [2/8] [========================100%] report4.sqlite [2/8] [========================100%] report4.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 63.4 596797662 16 37299853.9 11463331.5 1312 100128054 44505075.2 poll + 12.3 115657312 38 3043613.5 6326.5 3507 114185851 18517552.8 fopen + 12.0 112644556 591 190599.9 23554.0 1092 20946538 916359.7 ioctl + 9.5 89859962 6 14976660.3 107220.5 7223 81799661 32883386.3 fread + 0.9 8564355 1 8564355.0 8564355.0 8564355 8564355 0.0 pthread_cond_wait + 0.8 7100598 1 7100598.0 7100598.0 7100598 7100598 0.0 pthread_cond_broadcast + 0.5 4587263 32 143352.0 2204.5 1333 3746586 671302.8 fclose + 0.2 2342308 25 93692.3 9026.0 6772 1633214 323364.9 mmap64 + 0.2 2014448 10 201444.8 31454.0 22051 884024 304027.7 write + 0.1 593932 17 34937.2 8366.0 1873 379470 90751.4 mmap + 0.0 372607 2 186303.5 186303.5 168375 204232 25354.7 pthread_create + 0.0 320273 55 5823.1 6021.0 2334 10550 2072.2 open64 + 0.0 170338 1 170338.0 170338.0 170338 170338 0.0 fflush + 0.0 77535 1 77535.0 77535.0 77535 77535 0.0 fgets + 0.0 60311 8 7538.9 4633.5 2735 23083 6839.3 munmap + 0.0 37641 6 6273.5 6372.0 3216 9387 2126.0 open + 0.0 22875 3 7625.0 9138.0 3808 9929 3329.2 pipe2 + 0.0 17262 9 1918.0 1423.0 1022 3747 993.8 read + 0.0 16491 2 8245.5 8245.5 5891 10600 3329.8 socket + 0.0 10971 1 10971.0 10971.0 10971 10971 0.0 connect + 0.0 6632 1 6632.0 6632.0 6632 6632 0.0 fopen64 + 0.0 6061 1 6061.0 6061.0 6061 6061 0.0 pthread_mutex_trylock + 0.0 3176 2 1588.0 1588.0 1453 1723 190.9 fwrite + 0.0 2365 1 2365.0 2365.0 2365 2365 0.0 bind + 0.0 1042 1 1042.0 1042.0 1042 1042 0.0 fcntl + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- -------- ----------- --------------------------------- + 87.1 91151498 3 30383832.7 36093633.0 21130 55036735 27948713.1 cudaMemcpy + 5.1 5373556 12 447796.3 6998.0 200 1853236 632488.6 cudaFree + 2.3 2415988 1 2415988.0 2415988.0 2415988 2415988 0.0 cudaFreeHost + 1.7 1811777 1 1811777.0 1811777.0 1811777 1811777 0.0 cudaMallocHost + 1.4 1447858 1 1447858.0 1447858.0 1447858 1447858 0.0 cudaGetDeviceProperties_v2_v12000 + 1.1 1146304 171 6703.5 2825.0 2715 449320 34539.6 cudaLaunchKernel + 1.0 1047369 6 174561.5 124498.0 3266 462335 182577.2 cudaMalloc + 0.1 147859 838 176.4 150.0 80 721 81.0 cuGetProcAddress_v2 + 0.0 46527 1 46527.0 46527.0 46527 46527 0.0 cudaMemset + 0.0 15610 18 867.2 301.0 250 5870 1417.2 cudaEventCreateWithFlags + 0.0 13797 4 3449.3 2790.5 1002 7214 2662.6 cudaDeviceSynchronize + 0.0 7324 18 406.9 270.5 201 2194 468.7 cudaEventDestroy + 0.0 5211 4 1302.8 1253.0 1242 1463 107.0 cuInit + 0.0 1252 3 417.3 130.0 110 1012 515.1 cuModuleGetLoadingMode + 0.0 862 2 431.0 431.0 281 581 212.1 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- -------- -------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 88.5 42371434 49 864723.1 794053.0 249378 11507721 1574324.4 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 8.5 4090620 25 163624.8 164129.0 158689 166433 2048.2 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 0.6 282529 12 23544.1 23536.0 22944 23968 352.1 attention_v_kernel(float *, float *, float *, int, int, int, int) + 0.6 278114 12 23176.2 23136.0 22848 23680 232.5 preattention_scores_kernel(float *, float *, int, int, int, int, float *) + 0.5 226402 12 18866.8 18864.0 18784 18945 50.2 gelu_forward_kernel(float *, const float *, int) + 0.5 215713 12 17976.1 17968.5 17856 18144 94.6 softmax_forward_kernel(float *, float, const float *, int, int) + 0.3 145216 24 6050.7 6064.0 5632 6496 364.6 residual_forward_kernel(float *, float *, float *, int) + 0.2 110113 12 9176.1 9200.0 8928 9472 142.4 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.2 85280 1 85280.0 85280.0 85280 85280 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.1 70369 12 5864.1 5856.0 5824 5920 27.6 unpermute_kernel(float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 81.7 35990437 2 17995218.5 17995218.5 864 35989573 25447860.2 [CUDA memcpy Host-to-Device] + 17.4 7673713 1 7673713.0 7673713.0 7673713 7673713 0.0 [CUDA memcpy Device-to-Host] + 0.9 407011 1 407011.0 407011.0 407011 407011 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/report4.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/report4.sqlite diff --git a/test_gpt2_nsys_prof_req_1.out b/test_gpt2_nsys_prof_req_1.out new file mode 100644 index 0000000..9483a17 --- /dev/null +++ b/test_gpt2_nsys_prof_req_1.out @@ -0,0 +1,108 @@ +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.379311 +-39.836346, -39.790001 +-43.065910, -43.020863 +-42.828045, -42.780563 +-43.529541, -43.479824 +-44.318398, -44.268772 +-41.227425, -41.177418 +-41.270760, -41.221939 +-42.541393, -42.488613 +-42.394997, -42.345592 +---------------- Logits OK! (max err 0.055688, RMSE 0.001083) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-c915.qdstrm' + [1/8] [0% ] report6.nsys-rep [1/8] [0% ] report6.nsys-rep [1/8] [==19% ] report6.nsys-rep [1/8] [==21% ] report6.nsys-rep [1/8] [===22% ] report6.nsys-rep [1/8] [======35% ] report6.nsys-rep [1/8] [=======36% ] report6.nsys-rep [1/8] [==========49% ] report6.nsys-rep [1/8] [===========50% ] report6.nsys-rep [1/8] [===========52% ] report6.nsys-rep [1/8] [===========53% ] report6.nsys-rep [1/8] [============54% ] report6.nsys-rep [1/8] [============55% ] report6.nsys-rep [1/8] [============56% ] report6.nsys-rep [1/8] [============57% ] report6.nsys-rep [1/8] [=============58% ] report6.nsys-rep [1/8] [=============59% ] report6.nsys-rep [1/8] [==================75% ] report6.nsys-rep [1/8] [======================91% ] report6.nsys-rep [1/8] [========================100%] report6.nsys-rep [1/8] [========================100%] report6.nsys-rep + [2/8] [0% ] report6.sqlite [2/8] [1% ] report6.sqlite [2/8] [2% ] report6.sqlite [2/8] [3% ] report6.sqlite [2/8] [4% ] report6.sqlite [2/8] [5% ] report6.sqlite [2/8] [6% ] report6.sqlite [2/8] [7% ] report6.sqlite [2/8] [8% ] report6.sqlite [2/8] [9% ] report6.sqlite [2/8] [10% ] report6.sqlite [2/8] [11% ] report6.sqlite [2/8] [12% ] report6.sqlite [2/8] [13% ] report6.sqlite [2/8] [14% ] report6.sqlite [2/8] [=15% ] report6.sqlite [2/8] [=16% ] report6.sqlite [2/8] [=17% ] report6.sqlite [2/8] [==18% ] report6.sqlite [2/8] [==19% ] report6.sqlite [2/8] [==20% ] report6.sqlite [2/8] [==21% ] report6.sqlite [2/8] [===22% ] report6.sqlite [2/8] [===23% ] report6.sqlite [2/8] [===24% ] report6.sqlite [2/8] [====25% ] report6.sqlite [2/8] [====26% ] report6.sqlite [2/8] [====27% ] report6.sqlite [2/8] [====28% ] report6.sqlite [2/8] [=====29% ] report6.sqlite [2/8] [=====30% ] report6.sqlite [2/8] [=====31% ] report6.sqlite [2/8] [=====32% ] report6.sqlite [2/8] [======33% ] report6.sqlite [2/8] [======34% ] report6.sqlite [2/8] [======35% ] report6.sqlite [2/8] [=======36% ] report6.sqlite [2/8] [=======37% ] report6.sqlite [2/8] [=======38% ] report6.sqlite [2/8] [=======39% ] report6.sqlite [2/8] [========40% ] report6.sqlite [2/8] [========41% ] report6.sqlite [2/8] [========42% ] report6.sqlite [2/8] [=========43% ] report6.sqlite [2/8] [=========44% ] report6.sqlite [2/8] [=========45% ] report6.sqlite [2/8] [=========46% ] report6.sqlite [2/8] [==========47% ] report6.sqlite [2/8] [==========48% ] report6.sqlite [2/8] [==========49% ] report6.sqlite [2/8] [===========50% ] report6.sqlite [2/8] [===========51% ] report6.sqlite [2/8] [===========52% ] report6.sqlite [2/8] [===========53% ] report6.sqlite [2/8] [============54% ] report6.sqlite [2/8] [============55% ] report6.sqlite [2/8] [============56% ] report6.sqlite [2/8] [============57% ] report6.sqlite [2/8] [=============58% ] report6.sqlite [2/8] [=============59% ] report6.sqlite [2/8] [=============60% ] report6.sqlite [2/8] [==============61% ] report6.sqlite [2/8] [==============62% ] report6.sqlite [2/8] [==============63% ] report6.sqlite [2/8] [==============64% ] report6.sqlite [2/8] [===============65% ] report6.sqlite [2/8] [===============66% ] report6.sqlite [2/8] [===============67% ] report6.sqlite [2/8] [================68% ] report6.sqlite [2/8] [================69% ] report6.sqlite [2/8] [================70% ] report6.sqlite [2/8] [================71% ] report6.sqlite [2/8] [=================72% ] report6.sqlite [2/8] [=================73% ] report6.sqlite [2/8] [=================74% ] report6.sqlite [2/8] [==================75% ] report6.sqlite [2/8] [==================76% ] report6.sqlite [2/8] [==================77% ] report6.sqlite [2/8] [==================78% ] report6.sqlite [2/8] [===================79% ] report6.sqlite [2/8] [===================80% ] report6.sqlite [2/8] [===================81% ] report6.sqlite [2/8] [===================82% ] report6.sqlite [2/8] [====================83% ] report6.sqlite [2/8] [====================84% ] report6.sqlite [2/8] [====================85% ] report6.sqlite [2/8] [=====================86% ] report6.sqlite [2/8] [=====================87% ] report6.sqlite [2/8] [=====================88% ] report6.sqlite [2/8] [=====================89% ] report6.sqlite [2/8] [======================90% ] report6.sqlite [2/8] [======================91% ] report6.sqlite [2/8] [======================92% ] report6.sqlite [2/8] [=======================93% ] report6.sqlite [2/8] [=======================94% ] report6.sqlite [2/8] [=======================95% ] report6.sqlite [2/8] [=======================96% ] report6.sqlite [2/8] [========================97% ] report6.sqlite [2/8] [========================98% ] report6.sqlite [2/8] [========================99% ] report6.sqlite [2/8] [========================100%] report6.sqlite [2/8] [========================100%] report6.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ----------- ----------- -------- ---------- ----------- ---------------------- + 53.4 2194351837 32 68573494.9 100124295.0 1886 100179994 44439732.4 poll + 42.9 1763348095 7 251906870.7 58081562.0 1467 1200215470 444651839.0 fread + 2.9 119818701 600 199697.8 24235.0 1886 22988850 990232.3 ioctl + 0.5 21655089 38 569870.8 7543.0 4819 20367773 3300181.2 fopen + 0.1 2467848 25 98713.9 10057.0 7124 1710695 338493.4 mmap64 + 0.0 1330620 11 120965.5 45816.0 2375 925189 267079.5 write + 0.0 871902 32 27246.9 3736.5 2374 383499 92136.2 fclose + 0.0 634441 17 37320.1 9009.0 3353 404242 96596.2 mmap + 0.0 389366 2 194683.0 194683.0 194648 194718 49.5 pthread_create + 0.0 386078 55 7019.6 7193.0 3422 11105 1929.7 open64 + 0.0 135356 32 4229.9 1397.0 1396 91283 15885.4 fgets + 0.0 107831 56 1925.6 1885.0 1397 12781 1497.9 fcntl + 0.0 63625 2 31812.5 31812.5 1396 62229 43015.4 fflush + 0.0 62369 9 6929.9 6635.0 4330 10127 1985.0 munmap + 0.0 49378 6 8229.7 8171.5 4680 12502 2769.2 open + 0.0 49098 34 1444.1 1431.5 1396 1886 85.5 pthread_mutex_trylock + 0.0 41764 15 2784.3 2374.0 1886 5169 1140.3 read + 0.0 23746 3 7915.3 8940.0 4819 9987 2732.1 pipe2 + 0.0 21929 2 10964.5 10964.5 8660 13269 3259.1 socket + 0.0 11105 1 11105.0 11105.0 11105 11105 0.0 connect + 0.0 10965 7 1566.4 1467.0 1396 1886 220.6 dup + 0.0 7613 1 7613.0 7613.0 7613 7613 0.0 fopen64 + 0.0 6216 2 3108.0 3108.0 2933 3283 247.5 fwrite + 0.0 4190 3 1396.7 1397.0 1396 1397 0.6 pthread_cond_broadcast + 0.0 3841 1 3841.0 3841.0 3841 3841 0.0 bind + 0.0 1885 1 1885.0 1885.0 1885 1885 0.0 listen + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- -------- ----------- --------------------------------- + 73.6 59403870 3 19801290.0 22548780.0 21232 36833858 18559468.9 cudaMemcpy + 14.8 11937011 12 994750.9 11280.0 1397 7337536 2110925.3 cudaFree + 3.0 2437533 1 2437533.0 2437533.0 2437533 2437533 0.0 cudaFreeHost + 2.4 1975324 1 1975324.0 1975324.0 1975324 1975324 0.0 cudaGetDeviceProperties_v2_v12000 + 1.7 1380415 1 1380415.0 1380415.0 1380415 1380415 0.0 cudaMallocHost + 1.6 1294437 171 7569.8 4261.0 3702 400959 30590.6 cudaLaunchKernel + 1.4 1159422 838 1383.6 1397.0 489 50775 1808.0 cuGetProcAddress_v2 + 1.2 974146 6 162357.7 144885.5 4749 353537 148396.1 cudaMalloc + 0.1 52451 1 52451.0 52451.0 52451 52451 0.0 cudaMemset + 0.0 36526 18 2029.2 1397.0 1327 7682 1573.8 cudaEventCreateWithFlags + 0.0 34502 4 8625.5 7228.5 4400 15645 4884.7 cudaDeviceSynchronize + 0.0 29613 18 1645.2 1466.0 978 4749 815.2 cudaEventDestroy + 0.0 9919 4 2479.8 2375.0 2305 2864 258.3 cuInit + 0.0 5238 3 1746.0 1397.0 978 2863 989.8 cuModuleGetLoadingMode + 0.0 3283 2 1641.5 1641.5 1397 1886 345.8 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- -------- -------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 67.6 10665692 49 217667.2 162496.0 77632 2419814 332250.3 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 25.9 4086409 25 163456.4 163681.0 158656 166305 1957.5 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 1.4 226592 12 18882.7 18880.0 18720 19008 64.7 gelu_forward_kernel(float *, const float *, int) + 1.4 215201 12 17933.4 17920.0 17856 18112 72.7 softmax_forward_kernel(float *, float, const float *, int, int) + 0.9 145057 24 6044.0 6064.0 5664 6560 372.4 residual_forward_kernel(float *, float *, float *, int) + 0.6 99136 12 8261.3 8224.0 8160 8512 105.5 attention_v_kernel(float *, float *, float *, int, int, int, int) + 0.6 95616 12 7968.0 7968.0 7904 8032 36.1 preattention_scores_kernel(float *, float *, int, int, int, int, float *) + 0.6 94912 12 7909.3 7888.0 7776 8128 121.2 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.6 87072 1 87072.0 87072.0 87072 87072 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.4 70528 12 5877.3 5872.0 5856 5920 24.9 unpermute_kernel(float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 81.9 36688662 2 18344331.0 18344331.0 896 36687766 25941534.6 [CUDA memcpy Host-to-Device] + 17.2 7687347 1 7687347.0 7687347.0 7687347 7687347 0.0 [CUDA memcpy Device-to-Host] + 0.9 407265 1 407265.0 407265.0 407265 407265 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/report6.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/report6.sqlite diff --git a/test_gpt2_nsys_prof_req_2.out b/test_gpt2_nsys_prof_req_2.out new file mode 100644 index 0000000..362e2ac --- /dev/null +++ b/test_gpt2_nsys_prof_req_2.out @@ -0,0 +1,114 @@ +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431801 +-39.836346, -39.836525 +-43.065910, -43.066086 +-42.828045, -42.828217 +-43.529541, -43.529724 +-44.318398, -44.318581 +-41.227425, -41.227596 +-41.270760, -41.270927 +-42.541393, -42.541595 +-42.394997, -42.395184 +---------------- Logits OK! (max err 0.000069, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-e25f.qdstrm' + [1/8] [0% ] report7.nsys-rep [1/8] [0% ] report7.nsys-rep [1/8] [=16% ] report7.nsys-rep [1/8] [=====31% ] report7.nsys-rep [1/8] [=====32% ] report7.nsys-rep [1/8] [====25% ] report7.nsys-rep [1/8] [======35% ] report7.nsys-rep [1/8] [=======36% ] report7.nsys-rep [1/8] [=======37% ] report7.nsys-rep [1/8] [=======38% ] report7.nsys-rep [1/8] [=======39% ] report7.nsys-rep [1/8] [========40% ] report7.nsys-rep [1/8] [========41% ] report7.nsys-rep [1/8] [========42% ] report7.nsys-rep [1/8] [==================78% ] report7.nsys-rep [1/8] [=====================86% ] report7.nsys-rep [1/8] [========================100%] report7.nsys-rep [1/8] [========================100%] report7.nsys-rep + [2/8] [0% ] report7.sqlite [2/8] [1% ] report7.sqlite [2/8] [2% ] report7.sqlite [2/8] [3% ] report7.sqlite [2/8] [4% ] report7.sqlite [2/8] [5% ] report7.sqlite [2/8] [6% ] report7.sqlite [2/8] [7% ] report7.sqlite [2/8] [8% ] report7.sqlite [2/8] [9% ] report7.sqlite [2/8] [10% ] report7.sqlite [2/8] [11% ] report7.sqlite [2/8] [12% ] report7.sqlite [2/8] [13% ] report7.sqlite [2/8] [14% ] report7.sqlite [2/8] [=15% ] report7.sqlite [2/8] [=16% ] report7.sqlite [2/8] [=17% ] report7.sqlite [2/8] [==18% ] report7.sqlite [2/8] [==19% ] report7.sqlite [2/8] [==20% ] report7.sqlite [2/8] [==21% ] report7.sqlite [2/8] [===22% ] report7.sqlite [2/8] [===23% ] report7.sqlite [2/8] [===24% ] report7.sqlite [2/8] [====25% ] report7.sqlite [2/8] [====26% ] report7.sqlite [2/8] [====27% ] report7.sqlite [2/8] [====28% ] report7.sqlite [2/8] [=====29% ] report7.sqlite [2/8] [=====30% ] report7.sqlite [2/8] [=====31% ] report7.sqlite [2/8] [=====32% ] report7.sqlite [2/8] [======33% ] report7.sqlite [2/8] [======34% ] report7.sqlite [2/8] [======35% ] report7.sqlite [2/8] [=======36% ] report7.sqlite [2/8] [=======37% ] report7.sqlite [2/8] [=======38% ] report7.sqlite [2/8] [=======39% ] report7.sqlite [2/8] [========40% ] report7.sqlite [2/8] [========41% ] report7.sqlite [2/8] [========42% ] report7.sqlite [2/8] [=========43% ] report7.sqlite [2/8] [=========44% ] report7.sqlite [2/8] [=========45% ] report7.sqlite [2/8] [=========46% ] report7.sqlite [2/8] [==========47% ] report7.sqlite [2/8] [==========48% ] report7.sqlite [2/8] [==========49% ] report7.sqlite [2/8] [===========50% ] report7.sqlite [2/8] [===========51% ] report7.sqlite [2/8] [===========52% ] report7.sqlite [2/8] [===========53% ] report7.sqlite [2/8] [============54% ] report7.sqlite [2/8] [============55% ] report7.sqlite [2/8] [============56% ] report7.sqlite [2/8] [============57% ] report7.sqlite [2/8] [=============58% ] report7.sqlite [2/8] [=============59% ] report7.sqlite [2/8] [=============60% ] report7.sqlite [2/8] [==============61% ] report7.sqlite [2/8] [==============62% ] report7.sqlite [2/8] [==============63% ] report7.sqlite [2/8] [==============64% ] report7.sqlite [2/8] [===============65% ] report7.sqlite [2/8] [===============66% ] report7.sqlite [2/8] [===============67% ] report7.sqlite [2/8] [================68% ] report7.sqlite [2/8] [================69% ] report7.sqlite [2/8] [================70% ] report7.sqlite [2/8] [================71% ] report7.sqlite [2/8] [=================72% ] report7.sqlite [2/8] [=================73% ] report7.sqlite [2/8] [=================74% ] report7.sqlite [2/8] [==================75% ] report7.sqlite [2/8] [==================76% ] report7.sqlite [2/8] [==================77% ] report7.sqlite [2/8] [==================78% ] report7.sqlite [2/8] [===================79% ] report7.sqlite [2/8] [===================80% ] report7.sqlite [2/8] [===================81% ] report7.sqlite [2/8] [===================82% ] report7.sqlite [2/8] [====================83% ] report7.sqlite [2/8] [====================84% ] report7.sqlite [2/8] [====================85% ] report7.sqlite [2/8] [=====================86% ] report7.sqlite [2/8] [=====================87% ] report7.sqlite [2/8] [=====================88% ] report7.sqlite [2/8] [=====================89% ] report7.sqlite [2/8] [======================90% ] report7.sqlite [2/8] [======================91% ] report7.sqlite [2/8] [======================92% ] report7.sqlite [2/8] [=======================93% ] report7.sqlite [2/8] [=======================94% ] report7.sqlite [2/8] [=======================95% ] report7.sqlite [2/8] [=======================96% ] report7.sqlite [2/8] [========================97% ] report7.sqlite [2/8] [========================98% ] report7.sqlite [2/8] [========================99% ] report7.sqlite [2/8] [========================100%] report7.sqlite [2/8] [========================100%] report7.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- ---------------------- + 62.0 680717261 17 40042191.8 10065417.0 1532 100136863 44760103.1 poll + 14.8 162110818 38 4266074.2 6131.0 3496 161029552 26117864.7 fopen + 11.0 120373004 6 20062167.3 89457.5 6833 95159843 38120924.5 fread + 10.4 114685549 591 194053.4 23404.0 1032 22556281 980200.7 ioctl + 0.8 9116342 1 9116342.0 9116342.0 9116342 9116342 0.0 pthread_cond_broadcast + 0.4 3892842 32 121651.3 2144.0 1353 3504445 619781.4 fclose + 0.2 2358791 25 94351.6 8787.0 6392 1637323 324262.9 mmap64 + 0.2 1996705 1 1996705.0 1996705.0 1996705 1996705 0.0 pthread_cond_wait + 0.2 1950609 10 195060.9 27181.0 22051 798313 285177.5 write + 0.1 591677 17 34804.5 5981.0 1893 395630 94748.6 mmap + 0.0 362288 2 181144.0 181144.0 171561 190727 13552.4 pthread_create + 0.0 322858 55 5870.1 5861.0 2314 14327 2395.2 open64 + 0.0 76734 1 76734.0 76734.0 76734 76734 0.0 fgets + 0.0 62457 1 62457.0 62457.0 62457 62457 0.0 fflush + 0.0 40205 6 6700.8 6922.5 3527 9608 2269.3 open + 0.0 38922 8 4865.3 5074.5 2765 7153 1707.3 munmap + 0.0 19787 3 6595.7 7844.0 3657 8286 2554.5 pipe2 + 0.0 18904 10 1890.4 1623.0 1012 3516 882.0 read + 0.0 15960 2 7980.0 7980.0 5100 10860 4072.9 socket + 0.0 11101 1 11101.0 11101.0 11101 11101 0.0 connect + 0.0 6472 1 6472.0 6472.0 6472 6472 0.0 fopen64 + 0.0 3417 2 1708.5 1708.5 1393 2024 446.2 fwrite + 0.0 1593 1 1593.0 1593.0 1593 1593 0.0 bind + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- -------- ----------- --------------------------------- + 61.4 51907511 3 17302503.7 15637119.0 21119 36249273 18171403.8 cudaMemcpy + 18.9 15960434 4 3990108.5 3249843.5 1031059 8429688 3147630.5 cuLibraryLoadData + 6.1 5177442 12 431453.5 6161.5 191 1837587 621479.2 cudaFree + 5.7 4819182 1347 3577.7 3086.0 2835 335788 9243.6 cudaLaunchKernel + 2.9 2489386 1 2489386.0 2489386.0 2489386 2489386 0.0 cudaFreeHost + 1.7 1448309 1 1448309.0 1448309.0 1448309 1448309 0.0 cudaGetDeviceProperties_v2_v12000 + 1.6 1383518 1 1383518.0 1383518.0 1383518 1383518 0.0 cudaMallocHost + 0.9 766724 6 127787.3 123215.5 3447 318305 118859.5 cudaMalloc + 0.2 160077 838 191.0 151.0 70 6683 242.2 cuGetProcAddress_v2 + 0.2 139420 48 2904.6 2500.0 2184 13164 1581.6 cudaMemsetAsync + 0.2 137165 48 2857.6 2690.0 1974 12383 1519.2 cudaEventRecord + 0.0 35015 1 35015.0 35015.0 35015 35015 0.0 cudaMemset + 0.0 30347 4 7586.8 7734.0 6913 7966 469.4 cuLibraryGetKernel + 0.0 27345 240 113.9 90.0 69 1262 81.3 cuStreamGetCaptureInfo_v2 + 0.0 17219 18 956.6 300.0 260 7834 1819.2 cudaEventCreateWithFlags + 0.0 12852 4 3213.0 2549.5 991 6762 2485.5 cudaDeviceSynchronize + 0.0 8116 18 450.9 301.0 211 2675 566.0 cudaEventDestroy + 0.0 5371 4 1342.8 1343.0 1172 1513 139.5 cuInit + 0.0 3317 1 3317.0 3317.0 3317 3317 0.0 cudaEventQuery + 0.0 1322 3 440.7 180.0 130 1012 495.4 cuModuleGetLoadingMode + 0.0 872 2 436.0 436.0 211 661 318.2 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- -------- -------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 25.6 4083613 25 163344.5 163744.0 158464 166208 2004.3 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 22.8 3644222 576 6326.8 6336.0 6144 6592 79.1 ampere_sgemm_32x128_nn + 16.8 2675550 25 107022.0 78560.0 60384 1000383 186333.0 ampere_sgemm_128x64_tn + 15.9 2535902 576 4402.6 4384.0 4352 4864 60.7 ampere_sgemm_32x32_sliced1x4_tn + 10.0 1591582 12 132631.8 133168.0 129568 134783 1430.5 ampere_sgemm_64x64_tn + 2.4 390976 12 32581.3 32528.0 32384 32928 170.9 ampere_sgemm_64x32_sliced1x4_tn + 1.4 223072 12 18589.3 18560.0 18528 18848 90.0 gelu_forward_kernel(float *, const float *, int) + 1.4 219968 48 4582.7 4288.0 2848 7616 1622.8 bias_add_kernel(float *, const float *, int, int) + 1.4 217344 12 18112.0 18000.0 17920 19200 353.5 softmax_forward_kernel(float *, float, const float *, int, int) + 0.9 142080 24 5920.0 5952.0 5632 6240 265.2 residual_forward_kernel(float *, float *, float *, int) + 0.6 89376 1 89376.0 89376.0 89376 89376 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.5 84511 12 7042.6 7008.0 6976 7392 112.9 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.4 70944 12 5912.0 5888.0 5824 6016 62.7 unpermute_kernel(float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 81.3 36158470 2 18079235.0 18079235.0 832 36157638 25566722.7 [CUDA memcpy Host-to-Device] + 17.6 7821051 1 7821051.0 7821051.0 7821051 7821051 0.0 [CUDA memcpy Device-to-Host] + 1.1 473504 49 9663.3 1184.0 480 407616 58036.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.407 49 4.743 0.000 0.000 232.392 33.199 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/report7.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/report7.sqlite diff --git a/test_gpt2_nsys_prof_req_3.out b/test_gpt2_nsys_prof_req_3.out new file mode 100644 index 0000000..873d9ca --- /dev/null +++ b/test_gpt2_nsys_prof_req_3.out @@ -0,0 +1,106 @@ +test-gpt2-kernels - TF32 Mode is turned OFF. +test-gpt2 - [State] +Batch Size: 4 +Sequence Length: 64 +GPT2 - Allocated 221 MiB for activations +test-gpt2 - Printing 10 Expected vs Actual Logits: +-43.431618, -43.431660 +-39.836346, -39.836384 +-43.065910, -43.065952 +-42.828045, -42.828083 +-43.529541, -43.529587 +-44.318398, -44.318439 +-41.227425, -41.227470 +-41.270760, -41.270802 +-42.541393, -42.541447 +-42.394997, -42.395035 +---------------- Logits OK! (max err 0.000088, RMSE 0.000002) ---------------- +_________________________ All tests passed! _________________________ +Collecting data... +Generating '/tmp/nsys-report-b36a.qdstrm' + [1/8] [0% ] report5.nsys-rep [1/8] [0% ] report5.nsys-rep [1/8] [====25% ] report5.nsys-rep [1/8] [=========46% ] report5.nsys-rep [1/8] [==========48% ] report5.nsys-rep [1/8] [===========50% ] report5.nsys-rep [1/8] [===========51% ] report5.nsys-rep [1/8] [======34% ] report5.nsys-rep [1/8] [===========50% ] report5.nsys-rep [1/8] [===========51% ] report5.nsys-rep [1/8] [===========52% ] report5.nsys-rep [1/8] [===========53% ] report5.nsys-rep [1/8] [============54% ] report5.nsys-rep [1/8] [============55% ] report5.nsys-rep [1/8] [============56% ] report5.nsys-rep [1/8] [============57% ] report5.nsys-rep [1/8] [=============58% ] report5.nsys-rep [1/8] [=============59% ] report5.nsys-rep [1/8] [==================77% ] report5.nsys-rep [1/8] [======================91% ] report5.nsys-rep [1/8] [========================100%] report5.nsys-rep [1/8] [========================100%] report5.nsys-rep + [2/8] [0% ] report5.sqlite [2/8] [1% ] report5.sqlite [2/8] [2% ] report5.sqlite [2/8] [3% ] report5.sqlite [2/8] [4% ] report5.sqlite [2/8] [5% ] report5.sqlite [2/8] [6% ] report5.sqlite [2/8] [7% ] report5.sqlite [2/8] [8% ] report5.sqlite [2/8] [9% ] report5.sqlite [2/8] [10% ] report5.sqlite [2/8] [11% ] report5.sqlite [2/8] [12% ] report5.sqlite [2/8] [13% ] report5.sqlite [2/8] [14% ] report5.sqlite [2/8] [=15% ] report5.sqlite [2/8] [=16% ] report5.sqlite [2/8] [=17% ] report5.sqlite [2/8] [==18% ] report5.sqlite [2/8] [==19% ] report5.sqlite [2/8] [==20% ] report5.sqlite [2/8] [==21% ] report5.sqlite [2/8] [===22% ] report5.sqlite [2/8] [===23% ] report5.sqlite [2/8] [===24% ] report5.sqlite [2/8] [====25% ] report5.sqlite [2/8] [====26% ] report5.sqlite [2/8] [====27% ] report5.sqlite [2/8] [====28% ] report5.sqlite [2/8] [=====29% ] report5.sqlite [2/8] [=====30% ] report5.sqlite [2/8] [=====31% ] report5.sqlite [2/8] [=====32% ] report5.sqlite [2/8] [======33% ] report5.sqlite [2/8] [======34% ] report5.sqlite [2/8] [======35% ] report5.sqlite [2/8] [=======36% ] report5.sqlite [2/8] [=======37% ] report5.sqlite [2/8] [=======38% ] report5.sqlite [2/8] [=======39% ] report5.sqlite [2/8] [========40% ] report5.sqlite [2/8] [========41% ] report5.sqlite [2/8] [========42% ] report5.sqlite [2/8] [=========43% ] report5.sqlite [2/8] [=========44% ] report5.sqlite [2/8] [=========45% ] report5.sqlite [2/8] [=========46% ] report5.sqlite [2/8] [==========47% ] report5.sqlite [2/8] [==========48% ] report5.sqlite [2/8] [==========49% ] report5.sqlite [2/8] [===========50% ] report5.sqlite [2/8] [===========51% ] report5.sqlite [2/8] [===========52% ] report5.sqlite [2/8] [===========53% ] report5.sqlite [2/8] [============54% ] report5.sqlite [2/8] [============55% ] report5.sqlite [2/8] [============56% ] report5.sqlite [2/8] [============57% ] report5.sqlite [2/8] [=============58% ] report5.sqlite [2/8] [=============59% ] report5.sqlite [2/8] [=============60% ] report5.sqlite [2/8] [==============61% ] report5.sqlite [2/8] [==============62% ] report5.sqlite [2/8] [==============63% ] report5.sqlite [2/8] [==============64% ] report5.sqlite [2/8] [===============65% ] report5.sqlite [2/8] [===============66% ] report5.sqlite [2/8] [===============67% ] report5.sqlite [2/8] [================68% ] report5.sqlite [2/8] [================69% ] report5.sqlite [2/8] [================70% ] report5.sqlite [2/8] [================71% ] report5.sqlite [2/8] [=================72% ] report5.sqlite [2/8] [=================73% ] report5.sqlite [2/8] [=================74% ] report5.sqlite [2/8] [==================75% ] report5.sqlite [2/8] [==================76% ] report5.sqlite [2/8] [==================77% ] report5.sqlite [2/8] [==================78% ] report5.sqlite [2/8] [===================79% ] report5.sqlite [2/8] [===================80% ] report5.sqlite [2/8] [===================81% ] report5.sqlite [2/8] [===================82% ] report5.sqlite [2/8] [====================83% ] report5.sqlite [2/8] [====================84% ] report5.sqlite [2/8] [====================85% ] report5.sqlite [2/8] [=====================86% ] report5.sqlite [2/8] [=====================87% ] report5.sqlite [2/8] [=====================88% ] report5.sqlite [2/8] [=====================89% ] report5.sqlite [2/8] [======================90% ] report5.sqlite [2/8] [======================91% ] report5.sqlite [2/8] [======================92% ] report5.sqlite [2/8] [=======================93% ] report5.sqlite [2/8] [=======================94% ] report5.sqlite [2/8] [=======================95% ] report5.sqlite [2/8] [=======================96% ] report5.sqlite [2/8] [========================97% ] report5.sqlite [2/8] [========================98% ] report5.sqlite [2/8] [========================99% ] report5.sqlite [2/8] [========================100%] report5.sqlite [2/8] [========================100%] report5.sqlite +[3/8] Executing 'nvtx_sum' stats report +[4/8] Executing 'osrt_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- --------- -------- --------- ----------- ---------------------- + 72.8 691449484 17 40673499.1 9775260.0 1583 100131438 45632127.3 poll + 12.5 118880952 591 201152.2 23013.0 1032 24864504 1071706.0 ioctl + 10.3 97973444 6 16328907.3 63448.5 8215 89350283 35931749.3 fread + 1.5 14522344 38 382166.9 6252.0 3516 8002510 1594225.9 fopen + 0.9 8531381 1 8531381.0 8531381.0 8531381 8531381 0.0 pthread_cond_wait + 0.7 7052981 1 7052981.0 7052981.0 7052981 7052981 0.0 pthread_cond_broadcast + 0.5 4782327 32 149447.7 2219.0 1342 4255792 753563.6 fclose + 0.2 2370702 25 94828.1 8777.0 5971 1658446 328407.0 mmap64 + 0.2 2034120 11 184920.0 25638.0 1142 842979 290924.4 write + 0.1 633095 17 37240.9 4338.0 1904 388908 93374.4 mmap + 0.0 365446 2 182723.0 182723.0 177223 188223 7778.2 pthread_create + 0.0 330916 55 6016.7 5962.0 2335 10970 2241.2 open64 + 0.0 86322 1 86322.0 86322.0 86322 86322 0.0 fgets + 0.0 58760 1 58760.0 58760.0 58760 58760 0.0 fflush + 0.0 41479 8 5184.9 4508.5 2685 11542 2877.0 munmap + 0.0 40978 6 6829.7 6848.0 3146 9859 2589.1 open + 0.0 23634 3 7878.0 9628.0 4248 9758 3144.3 pipe2 + 0.0 18855 2 9427.5 9427.5 7614 11241 2564.7 socket + 0.0 18135 9 2015.0 1904.0 1012 3897 1071.0 read + 0.0 9648 1 9648.0 9648.0 9648 9648 0.0 connect + 0.0 6322 1 6322.0 6322.0 6322 6322 0.0 fopen64 + 0.0 3285 2 1642.5 1642.5 1372 1913 382.5 fwrite + 0.0 2184 1 2184.0 2184.0 2184 2184 0.0 bind + 0.0 1042 1 1042.0 1042.0 1042 1042 0.0 listen + +[5/8] Executing 'cuda_api_sum' stats report + + Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- ---------- ---------- -------- --------- ----------- --------------------------------- + 95.3 290852270 3 96950756.7 37077310.0 19757 253755203 137054884.4 cudaMemcpy + 2.0 6229529 12 519127.4 5315.0 191 2251928 798535.2 cudaFree + 0.8 2467101 1 2467101.0 2467101.0 2467101 2467101 0.0 cudaFreeHost + 0.6 1704733 1 1704733.0 1704733.0 1704733 1704733 0.0 cudaGetDeviceProperties_v2_v12000 + 0.5 1553469 1 1553469.0 1553469.0 1553469 1553469 0.0 cudaMallocHost + 0.3 1039319 171 6077.9 2826.0 2705 389279 29892.9 cudaLaunchKernel + 0.3 938599 6 156433.2 140363.0 3266 449833 164344.2 cudaMalloc + 0.1 173923 838 207.5 181.0 90 4588 181.4 cuGetProcAddress_v2 + 0.0 42730 1 42730.0 42730.0 42730 42730 0.0 cudaMemset + 0.0 13035 4 3258.8 3111.0 1072 5741 1916.5 cudaDeviceSynchronize + 0.0 12044 18 669.1 291.0 260 2676 789.3 cudaEventCreateWithFlags + 0.0 6834 18 379.7 270.5 210 1814 374.7 cudaEventDestroy + 0.0 5199 4 1299.8 1317.5 1202 1362 69.6 cuInit + 0.0 1433 3 477.7 120.0 110 1203 628.2 cuModuleGetLoadingMode + 0.0 1233 2 616.5 616.5 481 752 191.6 cudaGetDriverEntryPoint_v11030 + +[6/8] Executing 'cuda_gpu_kern_sum' stats report + + Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name + -------- --------------- --------- --------- --------- -------- -------- ----------- ---------------------------------------------------------------------------------------------------- + 99.0 243425601 49 4967869.4 4539787.0 1227363 73783097 10137706.5 matmul_forward_kernel(float *, const float *, const float *, const float *, int, int, int, int) + 0.5 1135204 12 94600.3 94560.5 94273 95360 268.7 preattention_scores_kernel(float *, float *, int, int, int, int, float *) + 0.1 345442 12 28786.8 28752.5 28224 29696 465.2 attention_v_kernel(float *, float *, float *, int, int, int, int) + 0.1 226976 12 18914.7 18912.0 18688 19072 88.9 gelu_forward_kernel(float *, const float *, int) + 0.1 146081 24 6086.7 6112.0 5664 6625 369.0 residual_forward_kernel(float *, float *, float *, int) + 0.1 141856 12 11821.3 11808.0 11712 11968 88.9 softmax_forward_kernel(float *, float, const float *, int, int) + 0.1 131456 25 5258.2 5120.0 4704 6016 438.5 layernorm_forward_kernel(float *, float *, float *, const float *, const float *, const float *, in… + 0.0 117888 12 9824.0 9824.0 9664 10144 133.7 permute_kernel(float *, float *, float *, const float *, int, int, int, int) + 0.0 90304 1 90304.0 90304.0 90304 90304 0.0 encoder_forward_kernel(float *, const int *, const float *, const float *, int, int, int) + 0.0 70368 12 5864.0 5856.0 5792 5920 33.8 unpermute_kernel(float *, float *, int, int, int, int) + +[7/8] Executing 'cuda_gpu_mem_time_sum' stats report + + Time (%) Total Time (ns) Count Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Operation + -------- --------------- ----- ---------- ---------- -------- -------- ----------- ---------------------------- + 80.8 36866268 2 18433134.0 18433134.0 896 36865372 26067121.0 [CUDA memcpy Host-to-Device] + 18.3 8354069 1 8354069.0 8354069.0 8354069 8354069 0.0 [CUDA memcpy Device-to-Host] + 0.9 406945 1 406945.0 406945.0 406945 406945 0.0 [CUDA memset] + +[8/8] Executing 'cuda_gpu_mem_size_sum' stats report + + Total (MB) Count Avg (MB) Med (MB) Min (MB) Max (MB) StdDev (MB) Operation + ---------- ----- -------- -------- -------- -------- ----------- ---------------------------- + 497.905 2 248.952 248.952 0.001 497.904 352.070 [CUDA memcpy Host-to-Device] + 232.392 1 232.392 232.392 232.392 232.392 0.000 [CUDA memset] + 51.511 1 51.511 51.511 51.511 51.511 0.000 [CUDA memcpy Device-to-Host] + +Generated: + /u/asingh41/sp26_ece408_Temperature1.5/report5.nsys-rep + /u/asingh41/sp26_ece408_Temperature1.5/report5.sqlite diff --git a/test_op13.sh b/test_op13.sh new file mode 100644 index 0000000..72345a3 --- /dev/null +++ b/test_op13.sh @@ -0,0 +1,7 @@ +cp gpt2.cuh gpt2.cuh.bak_op13 +cp kernels_op_13/gpt2.cuh gpt2.cuh + +make test_gpt2 op_13 + +# revert host file +mv gpt2.cuh.bak_op13 gpt2.cuh \ No newline at end of file diff --git a/test_op16.sh b/test_op16.sh new file mode 100644 index 0000000..45b77e0 --- /dev/null +++ b/test_op16.sh @@ -0,0 +1,14 @@ +cp gpt2.cuh gpt2.cuh.bak_op16 +cp kernels_op_16/gpt2.cuh gpt2.cuh + + +cp kernels/matmul.cuh matmul.cuh.bak_op16 +cp kernels_op_16/matmul.cuh kernels/matmul.cuh + + +make test_gpt2 op_16 + + +# revert host file +mv gpt2.cuh.bak_op16 gpt2.cuh +mv matmul.cuh.bak_op16 kernels/matmul.cuh \ No newline at end of file diff --git a/test_op9.sh b/test_op9.sh new file mode 100644 index 0000000..d5db043 --- /dev/null +++ b/test_op9.sh @@ -0,0 +1,14 @@ +set -e +cp kernels_op_9/attention.cuh kernels/attention.cuh +cp kernels_op_9/encoder.cuh kernels/encoder.cuh +cp kernels_op_9/gelu.cuh kernels/gelu.cuh +cp kernels_op_9/layernorm.cuh kernels/layernorm.cuh +cp kernels_op_9/matmul.cuh kernels/matmul.cuh +cp kernels_op_9/residual.cuh kernels/residual.cuh +cp kernels_op_9/softmax.cuh kernels/softmax.cuh + +make test_gpt2 op9 + +git checkout kernels/attention.cuh kernels/encoder.cuh kernels/gelu.cuh \ + kernels/layernorm.cuh kernels/matmul.cuh kernels/residual.cuh \ + kernels/softmax.cuh \ No newline at end of file diff --git a/test_req0.sh b/test_req0.sh new file mode 100644 index 0000000..21de644 --- /dev/null +++ b/test_req0.sh @@ -0,0 +1,7 @@ +cp kernels_req_0/attention.cuh kernels/attention.cuh +cp kernels_req_0/matmul.cuh kernels/matmul.cuh + +make test_gpt2 + +git checkout kernels/attention.cuh +git checkout kernels/matmul.cuh \ No newline at end of file diff --git a/test_req1.sh b/test_req1.sh new file mode 100644 index 0000000..8f709c3 --- /dev/null +++ b/test_req1.sh @@ -0,0 +1,11 @@ +cp kernels_req_1/attention.cuh kernels/attention.cuh +cp kernels_req_1/matmul.cuh kernels/matmul.cuh + + + +make test_gpt2 + + + +git checkout kernels/attention.cuh +git checkout kernels/matmul.cuh \ No newline at end of file diff --git a/test_req2.sh b/test_req2.sh new file mode 100644 index 0000000..5f22d18 --- /dev/null +++ b/test_req2.sh @@ -0,0 +1,11 @@ +cp kernels_req_2/attention.cuh kernels/attention.cuh +cp kernels_req_2/matmul.cuh kernels/matmul.cuh + + + +make test_gpt2 + + + +git checkout kernels/attention.cuh +git checkout kernels/matmul.cuh \ No newline at end of file diff --git a/test_req3.sh b/test_req3.sh new file mode 100644 index 0000000..b173adf --- /dev/null +++ b/test_req3.sh @@ -0,0 +1,9 @@ +cp kernels_req_3/layernorm.cuh kernels/layernorm.cuh +cp kernels_req_3/softmax.cuh kernels/softmax.cuh +cp kernels_req_3/attention.cuh kernels/attention.cuh + +make test_gpt2 + +git checkout kernels/softmax.cuh +git checkout kernels/layernorm.cuh +git checkout kernels/attention.cuh \ No newline at end of file diff --git a/test_req6.sh b/test_req6.sh new file mode 100644 index 0000000..e9162ed --- /dev/null +++ b/test_req6.sh @@ -0,0 +1,14 @@ +cp kernels/attention.cuh "/u/asingh41/sp26_ece408_Temperature1.5/attention.cuh" + + +cp kernels_req_6/attention.cuh "/u/asingh41/sp26_ece408_Temperature1.5/kernels/attention.cuh" + + +make verify_kv_cache + + +cp attention.cuh kernels/attention.cuh + + + +rm attention.cuh \ No newline at end of file diff --git a/utils/cuda_utils.cuh b/utils/cuda_utils.cuh new file mode 100644 index 0000000..d3bb9e7 --- /dev/null +++ b/utils/cuda_utils.cuh @@ -0,0 +1,48 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +/* + Helpful CUDA utilities for error checking and other common operations. +*/ +#ifndef CUDA_UTILS_CUH +#define CUDA_UTILS_CUH + +#include +#include +#include +#include + +#include +#include + +// convenience macro for calculating grid/block dimensions for kernels +#define CEIL_DIV(M, N) (((M) + (N)-1) / (N)) + +// CUDA error checking +void cudaCheck(cudaError_t error, const char *file, int line) { + if (error != cudaSuccess) { + printf("[CUDA ERROR] at file %s:%d:\n%s\n", file, line, + cudaGetErrorString(error)); + exit(EXIT_FAILURE); + } +}; +#define cudaCheck(err) (cudaCheck(err, __FILE__, __LINE__)) + +// cuBLAS error checking +void cublasCheck(cublasStatus_t status, const char *file, int line) +{ + if (status != CUBLAS_STATUS_SUCCESS) { + printf("[cuBLAS ERROR]: %d %s %d\n", status, file, line); + exit(EXIT_FAILURE); + } +} +#define cublasCheck(status) { cublasCheck((status), __FILE__, __LINE__); } + +static cublasComputeType_t cublas_compute_type; +cublasHandle_t cublas_handle; + +#endif // CUDA_UTILS_CUH \ No newline at end of file diff --git a/utils/tokenizer.h b/utils/tokenizer.h new file mode 100644 index 0000000..49781f1 --- /dev/null +++ b/utils/tokenizer.h @@ -0,0 +1,108 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +/* +Defines the GPT-2 Tokenizer. +Only supports decoding, i.e.: tokens (integers) -> strings +*/ +#ifndef UTILS_TOKENIZER_H +#define UTILS_TOKENIZER_H + +#include +#include +#include +#include "utils.h" + +// ---------------------------------------------------------------------------- + +typedef struct { + uint32_t vocab_size; + char **token_table; + int init_ok; + int eot_token; // <|endoftext|> token id +} Tokenizer; + +void safe_printf(const char *piece) { + // the tokens are raw bytes, and we we only want to print the printable ones + // many bytes can be various control codes, backspace, etc. + if (piece == NULL) { return; } + if (piece[0] == '\0') { return; } + // handle individual byte tokens + // every token is asserted to be at least one byte so doing piece[1] is ok + if (piece[1] == '\0') { + unsigned char byte_val = piece[0]; + if (!(isprint(byte_val) || isspace(byte_val))) { + return; // weird byte, don't print it + } + } + printf("%s", piece); +} + +void tokenizer_init(Tokenizer *tokenizer, const char *filename) { + FILE *file = fopen(filename, "rb"); + if (file == NULL) { + printf("---\n"); + printf("Tokenizer - WARNING: Failed to open the tokenizer file %s\n", filename); + printf("---\n"); + tokenizer->init_ok = 0; + return; + } + // read in the header + uint32_t header[256]; + freadCheck(header, sizeof(uint32_t), 256, file); + assert(header[0] == 20240328); + int version = header[1]; + tokenizer->vocab_size = header[2]; + if (version == 1) { + // version 1 didn't include the EOT token id + // so we assume it is 50256, the EOT in GPT-2 + assert(tokenizer->vocab_size == 50257); // let's be defensive here + tokenizer->eot_token = 50256; + } else if (version == 2) { + tokenizer->eot_token = header[3]; + } else { + fprintf(stderr, "Tokenizer model file %s has bad version: %d\n", filename, version); + exit(EXIT_FAILURE); + } + // read in all the tokens + unsigned char length; + tokenizer->token_table = (char **)mallocCheck(tokenizer->vocab_size * sizeof(char *)); + for (uint32_t i = 0; i < tokenizer->vocab_size; i++) { + freadCheck(&length, sizeof(unsigned char), 1, file); + assert(length > 0); // every token should be at least one character + char *token_bytes = (char *)mallocCheck(length + 1); + freadCheck(token_bytes, sizeof(char), length, file); + token_bytes[length] = '\0'; // Add null terminator for printing + tokenizer->token_table[i] = token_bytes; + } + // cleanups + fcloseCheck(file); + tokenizer->init_ok = 1; +} + +const char *tokenizer_decode(Tokenizer *tokenizer, uint32_t token_id) { + if (tokenizer->init_ok == 0) { + return NULL; + } + if (token_id < tokenizer->vocab_size) { + return tokenizer->token_table[token_id]; + } else { + printf("Tokenizer - Invalid token id %u!\n", token_id); + return NULL; + } +} + +void tokenizer_free(Tokenizer *tokenizer) { + if (tokenizer->init_ok) { + for (uint32_t i = 0; i < tokenizer->vocab_size; i++) { + free(tokenizer->token_table[i]); + } + free(tokenizer->token_table); + } +} + +#endif // UTILS_TOKENIZER_H \ No newline at end of file diff --git a/utils/utils.h b/utils/utils.h new file mode 100644 index 0000000..70436c6 --- /dev/null +++ b/utils/utils.h @@ -0,0 +1,141 @@ +//----------------------------------------------------------------------------------------------- +// Copyright (c) 2024 Andrej Karpathy +// Licensed under the MIT License. See the LICENSE file for details. +// +// Modifications Copyright (c) 2025 Hanwen Liu, Hrishi Shah, Kelin Zeng, Charles Pei, and Vijay Daita, ALL RIGHTS RESERVED. +//----------------------------------------------------------------------------------------------- + +#ifndef UTILS_H +#define UTILS_H + +#include +#include +#include +#include +#ifndef _WIN32 +#include +#endif + +// ---------------------------------------------------------------------------- +// fread convenience utils, with nice handling of error checking using macros +// simple replace fopen, fread, fclose, fseek +// with fopenCheck, freadCheck, fcloseCheck, fseekCheck + +FILE *fopen_check(const char *path, const char *mode, const char *file, int line) { + FILE *fp = fopen(path, mode); + if (fp == NULL) { + fprintf(stderr, "Utils - Error: Failed to open file '%s' at %s:%d\n", path, file, line); + fprintf(stderr, "Error details:\n"); + fprintf(stderr, " File: %s\n", file); + fprintf(stderr, " Line: %d\n", line); + fprintf(stderr, " Path: %s\n", path); + fprintf(stderr, " Mode: %s\n", mode); + exit(EXIT_FAILURE); + } + return fp; +} + +#define fopenCheck(path, mode) fopen_check(path, mode, __FILE__, __LINE__) + +void fread_check(void *ptr, size_t size, size_t nmemb, FILE *stream, const char *file, int line) { + size_t result = fread(ptr, size, nmemb, stream); + if (result != nmemb) { + if (feof(stream)) { + fprintf(stderr, "Utils - Error: Unexpected end of file at %s:%d\n", file, line); + } else if (ferror(stream)) { + fprintf(stderr, "Utils - Error: File read error at %s:%d\n", file, line); + } else { + fprintf(stderr, "Utils - Error: Partial read at %s:%d. Expected %zu elements, read %zu\n", + file, line, nmemb, result); + } + fprintf(stderr, "Utils - Error details:\n"); + fprintf(stderr, " File: %s\n", file); + fprintf(stderr, " Line: %d\n", line); + fprintf(stderr, " Expected elements: %zu\n", nmemb); + fprintf(stderr, " Read elements: %zu\n", result); + exit(EXIT_FAILURE); + } +} + +#define freadCheck(ptr, size, nmemb, stream) fread_check(ptr, size, nmemb, stream, __FILE__, __LINE__) + +void fclose_check(FILE *fp, const char *file, int line) { + if (fclose(fp) != 0) { + fprintf(stderr, "Utils - Error: Failed to close file at %s:%d\n", file, line); + fprintf(stderr, "Error details:\n"); + fprintf(stderr, " File: %s\n", file); + fprintf(stderr, " Line: %d\n", line); + exit(EXIT_FAILURE); + } +} + +#define fcloseCheck(fp) fclose_check(fp, __FILE__, __LINE__) + +void fseek_check(FILE *fp, long off, int whence, const char *file, int line) { + if (fseek(fp, off, whence) != 0) { + fprintf(stderr, "Utils - Error: Failed to seek in file at %s:%d\n", file, line); + fprintf(stderr, "Error details:\n"); + fprintf(stderr, " Offset: %ld\n", off); + fprintf(stderr, " Whence: %d\n", whence); + fprintf(stderr, " File: %s\n", file); + fprintf(stderr, " Line: %d\n", line); + exit(EXIT_FAILURE); + } +} + +#define fseekCheck(fp, off, whence) fseek_check(fp, off, whence, __FILE__, __LINE__) + +// ---------------------------------------------------------------------------- +// malloc error-handling wrapper util + +void *malloc_check(size_t size, const char *file, int line) { + void *ptr = malloc(size); + if (ptr == NULL) { + fprintf(stderr, "Utils - Error: Memory allocation failed at %s:%d\n", file, line); + fprintf(stderr, "Error details:\n"); + fprintf(stderr, " File: %s\n", file); + fprintf(stderr, " Line: %d\n", line); + fprintf(stderr, " Size: %zu bytes\n", size); + exit(EXIT_FAILURE); + } + return ptr; +} + +#define mallocCheck(size) malloc_check(size, __FILE__, __LINE__) + +// ---------------------------------------------------------------------------- +// I/O ops + +void create_dir_if_not_exists(const char *dir) { + if (dir == NULL) { return; } + struct stat st = {0}; + if (stat(dir, &st) == -1) { + if (mkdir(dir, 0700) == -1) { + printf("Utils - ERROR: could not create directory: %s\n", dir); + exit(EXIT_FAILURE); + } + printf("Utils - Created directory: %s\n", dir); + } +} + +int find_max_step(const char* output_log_dir) { + // find the DONE file in the log dir with highest step count + if (output_log_dir == NULL) { return -1; } + DIR* dir; + struct dirent* entry; + int max_step = -1; + dir = opendir(output_log_dir); + if (dir == NULL) { return -1; } + while ((entry = readdir(dir)) != NULL) { + if (strncmp(entry->d_name, "DONE_", 5) == 0) { + int step = atoi(entry->d_name + 5); + if (step > max_step) { + max_step = step; + } + } + } + closedir(dir); + return max_step; +} + +#endif \ No newline at end of file diff --git a/verify.slurm b/verify.slurm new file mode 100644 index 0000000..8cac142 --- /dev/null +++ b/verify.slurm @@ -0,0 +1,30 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:25:00 + +module reset + +SUFFIX=${1:-op9} + +OUTDIR=output/simple +mkdir -p $OUTDIR + +OUTFILE=$OUTDIR/verify_${SUFFIX}.out +ERRFILE=$OUTDIR/verify_${SUFFIX}.err + +echo "Logging to $OUTFILE and $ERRFILE" + +exec > $OUTFILE 2> $ERRFILE + +echo -e "job $SLURM_JOBID is starting on $(hostname)\n\n" + +srun ./build/test_gpt2_${SUFFIX} \ No newline at end of file diff --git a/verify_kv_cache.cu b/verify_kv_cache.cu new file mode 100644 index 0000000..bc23cea --- /dev/null +++ b/verify_kv_cache.cu @@ -0,0 +1,421 @@ +#define BENCH +#include "gpt2.cuh" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Global variables for Option 1 stateful KV Cache +bool g_enable_kv_cache = false; +float* g_kv_cache = nullptr; +int g_layer_idx = 0; +bool g_is_prefill = true; +int g_current_pos = 0; + + +struct Args { + int batch = 1; + int prompt_len = 32; + int gen_len = 32; + unsigned int seed = 0; + bool random_mode = false; +}; + +static void usage(const char* prog) { + fprintf(stderr, + "Usage: %s [--batch N] [--prompt-len L] [--gen-len G] [--seed S] [--random]\n" + "Defaults: --batch 1 --prompt-len 32 --gen-len 32 --seed 0\n", + prog); +} + +static Args parse_args(int argc, char** argv) { + Args a; + for (int i = 1; i < argc; ++i) { + std::string k = argv[i]; + auto need = [&](int i){ if (i+1 >= argc) { usage(argv[0]); exit(EXIT_FAILURE);} }; + if (k == "--batch") { need(i); a.batch = atoi(argv[++i]); } + else if (k == "--prompt-len"){ need(i); a.prompt_len = atoi(argv[++i]); } + else if (k == "--gen-len"){ need(i); a.gen_len = atoi(argv[++i]); } + else if (k == "--seed") { need(i); a.seed = (unsigned)strtoul(argv[++i], nullptr, 10); } + else if (k == "--random") { a.random_mode = true; } + else { usage(argv[0]); exit(EXIT_FAILURE); } + } + return a; +} + +static void fill_random_prompts(int* buf, int B, int stride, int prompt_len, int vocab_size) { + for (int b = 0; b < B; ++b) { + for (int t = 0; t < prompt_len; ++t) { + buf[b * stride + t] = rand() % vocab_size; + } + } +} + +static int sample_token_argmax(const float* logits, int vocab_size) { + float max_logit = -FLT_MAX; + int best = 0; + for (int i = 0; i < vocab_size; ++i) { + if (logits[i] > max_logit) { + max_logit = logits[i]; + best = i; + } + } + return best; +} + +static void copy_last_logits(float* dest, const float* device_logits, int B, int cur_len, size_t Vp) { + for (int b = 0; b < B; ++b) { + size_t offset = (b * cur_len + (cur_len - 1)) * Vp; + cudaCheck(cudaMemcpy(dest + b * Vp, device_logits + offset, Vp * sizeof(float), cudaMemcpyDeviceToHost)); + } +} + +double benchmark_gpt2_forward(GPT2 *model, int* inputs, int B, int T) { + if (model->params_memory == NULL) { + printf("GPT2 - Error: model was not initialized properly.\n"); + exit(EXIT_FAILURE); + } + int V = model->config.vocab_size; + int Vp = model->config.padded_vocab_size; + int L = model->config.num_layers; + int NH = model->config.num_heads; + int C = model->config.channels; + + for(int i = 0; i < B * T; i++) { + assert(0 <= inputs[i] && inputs[i] < V); + } + + if(model->acts_memory == NULL || B != model->batch_size || T != model->seq_len) { + model->batch_size = B; + model->seq_len = T; + fill_in_activation_sizes(model->act_sizes, B, T, model->config); + size_t num_activations = 0; + for (size_t i = 0; i < NUM_ACTIVATION_TENSORS; i++) { + num_activations += model->act_sizes[i]; + } + model->num_activations = num_activations; + + if(model->acts_memory != NULL) { + cudaCheck(cudaFree(model->acts_memory)); + cudaCheck(cudaFreeHost(model->cpu_losses)); + cudaCheck(cudaFree(model->inputs)); + } + model->acts_memory = malloc_and_point_activations(&model->acts, model->act_sizes); + cudaCheck(cudaMalloc((void**)&model->inputs, B * T * sizeof(int))); + cudaCheck(cudaMallocHost((void**)&model->cpu_losses, B * T * sizeof(float))); + } + + cudaCheck(cudaMemcpy(model->inputs, inputs, B * T * sizeof(int), cudaMemcpyHostToDevice)); + + cudaCheck(cudaDeviceSynchronize()); + auto compute_start = std::chrono::high_resolution_clock::now(); + + ParameterTensors params = model->params; + ActivationTensors acts = model->acts; + float* residual; + + // When using KV cache in decode mode (T=1), we must use the correct + // absolute position for positional embeddings, not position 0. + if (g_enable_kv_cache && !g_is_prefill) { + // Decode: T=1, actual position is g_current_pos + // Use wpe offset to point to the correct position embedding + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe + g_current_pos * C, B, T, C); + } else { + encoder_forward(acts.encoded, model->inputs, params.wte, params.wpe, B, T, C); + } + + for (int l = 0; l < L; l++) { + residual = l == 0 ? acts.encoded : acts.residual3 + (l-1) * B * T * C; + + float* l_ln1w = params.ln1w + l * C; + float* l_ln1b = params.ln1b + l * C; + float* l_qkvw = params.qkvw + l * 3*C * C; + float* l_qkvb = params.qkvb + l * 3*C; + float* l_attprojw = params.attprojw + l * C * C; + float* l_attprojb = params.attprojb + l * C; + float* l_ln2w = params.ln2w + l * C; + float* l_ln2b = params.ln2b + l * C; + float* l_fcw = params.fcw + l * 4*C * C; + float* l_fcb = params.fcb + l * 4*C; + float* l_fcprojw = params.fcprojw + l * C * 4*C; + float* l_fcprojb = params.fcprojb + l * C; + + float* l_ln1 = acts.ln1 + l * B * T * C; + float* l_ln1_mean = acts.ln1_mean; + float* l_ln1_rstd = acts.ln1_rstd; + float* l_atty = acts.atty + l * B * T * C; + float* l_qkvr = acts.qkvr + l * B * T * 3*C; + float* l_att = acts.att + l * B * NH * T * T; + float* l_attproj = acts.attproj + l * B * T * C; + float* l_residual2 = acts.residual2 + l * B * T * C; + float* l_ln2 = acts.ln2 + l * B * T * C; + float* l_ln2_mean = acts.ln2_mean; + float* l_ln2_rstd = acts.ln2_rstd; + float* l_fch = acts.fch + l * B * T * 4*C; + float* l_fch_gelu = acts.fch_gelu + l * B * T * 4*C; + float* l_fcproj = acts.fcproj + l * B * T * C; + float* l_residual3 = acts.residual3 + l * B * T * C; + float* scratch = acts.output; + + layernorm_forward(l_ln1, l_ln1_mean, l_ln1_rstd, residual, l_ln1w, l_ln1b, B, T, C); + matmul_forward(scratch, l_ln1, l_qkvw, l_qkvb, B, T, C, 3*C); + attention_forward(l_atty, l_qkvr, l_att, scratch, B, T, C, NH); + matmul_forward(l_attproj, l_atty, l_attprojw, l_attprojb, B, T, C, C); + residual_forward(l_residual2, residual, l_attproj, B*T*C); + layernorm_forward(l_ln2, l_ln2_mean, l_ln2_rstd, l_residual2, l_ln2w, l_ln2b, B, T, C); + matmul_forward(l_fch, l_ln2, l_fcw, l_fcb, B, T, C, 4*C); + gelu_forward(l_fch_gelu, l_fch, B*T*4*C); + matmul_forward(l_fcproj, l_fch_gelu, l_fcprojw, l_fcprojb, B, T, 4*C, C); + residual_forward(l_residual3, l_residual2, l_fcproj, B*T*C); + } + + residual = acts.residual3 + (L-1) * B * T * C; + layernorm_forward(acts.lnf, acts.lnf_mean, acts.lnf_rstd, residual, params.lnfw, params.lnfb, B, T, C); + matmul_forward(acts.output, acts.lnf, params.wte, NULL, B, T, C, Vp); + + cudaCheck(cudaDeviceSynchronize()); + auto compute_end = std::chrono::high_resolution_clock::now(); + return std::chrono::duration(compute_end - compute_start).count(); +} + +static void copy_single_token_logits(float* dest, const float* device_logits, int B, size_t Vp) { + cudaCheck(cudaMemcpy(dest, device_logits, B * Vp * sizeof(float), cudaMemcpyDeviceToHost)); +} + +int main(int argc, char** argv) { + Args args = parse_args(argc, argv); + if (args.seed == 0) args.seed = time(nullptr); + srand(args.seed); + + int deviceIdx = 0; + cudaCheck(cudaSetDevice(deviceIdx)); + cudaDeviceProp deviceProp; + cudaGetDeviceProperties(&deviceProp, deviceIdx); + cublasCheck(cublasCreate(&cublas_handle)); + + GPT2 model; + gpt2_build_from_checkpoint(&model, "/work/hdd/bche/Project_GPT/gpt2_124M.bin"); + + Tokenizer tokenizer; + tokenizer_init(&tokenizer, "/work/hdd/bche/Project_GPT/gpt2_tokenizer.bin"); + + int B = args.batch; + int V = model.config.vocab_size; + int Vp = model.config.padded_vocab_size; + int maxT = model.config.max_seq_len; + int prompt_len = args.prompt_len; + int gen_len = args.gen_len; + + int longest_seq = prompt_len + gen_len; + if (longest_seq > maxT) { + fprintf(stderr, "Exceeds max sequence length %d\n", maxT); + return 1; + } + + int* sequence_buffer = (int*)mallocCheck(B * longest_seq * sizeof(int)); + if (args.random_mode) { + printf("[Prompt] Generating Random Prompts\n"); + fill_random_prompts(sequence_buffer, B, longest_seq, prompt_len, V); + } else { + printf("[Prompt] Loading Deterministic Prompts\n"); + FILE *state_file = fopenCheck("/work/hdd/bche/Project_GPT/gpt2_inference_verif.bin", "rb"); + int state_header[256]; + freadCheck(state_header, sizeof(int), 256, state_file); + int file_B = state_header[2]; + int file_T = state_header[3]; + B = file_B; + prompt_len = file_T; + longest_seq = prompt_len + gen_len; + + free(sequence_buffer); + sequence_buffer = (int*)mallocCheck(B * longest_seq * sizeof(int)); + int* input_tokens = (int*)mallocCheck(B * prompt_len * sizeof(int)); + freadCheck(input_tokens, sizeof(int), B * prompt_len, state_file); + + for (int b = 0; b < B; ++b) { + for (int t = 0; t < prompt_len; ++t) { + sequence_buffer[b * longest_seq + t] = input_tokens[b * prompt_len + t]; + } + } + free(input_tokens); + fcloseCheck(state_file); + } + + printf("[Config] B=%d, Prompt=%d, Gen=%d\n", B, prompt_len, gen_len); + + int* forward_inputs = (int*)mallocCheck(B * longest_seq * sizeof(int)); + float* baseline_logits = (float*)mallocCheck(gen_len * B * Vp * sizeof(float)); + std::vector baseline_latencies; + + // ================= BASELINE RUN ================= + printf("\n=== Running Baseline (No KV Cache) ===\n"); + g_enable_kv_cache = false; + int current_len = prompt_len; + + // Copy prefill prompt + for (int b = 0; b < B; ++b) { + memcpy(forward_inputs + b * current_len, sequence_buffer + b * longest_seq, current_len * sizeof(int)); + } + + // Prefill Baseline + double prefill_time = benchmark_gpt2_forward(&model, forward_inputs, B, current_len); + printf("Prefill latency: %.3f ms\n", prefill_time); + + for (int t = 0; t < gen_len; ++t) { + float* cur_logits = baseline_logits + t * B * Vp; + copy_last_logits(cur_logits, model.acts.output, B, current_len, Vp); + + for (int b = 0; b < B; ++b) { + sequence_buffer[b * longest_seq + current_len] = sample_token_argmax(cur_logits + b * Vp, V); + } + current_len++; + + // Baseline step + for (int b = 0; b < B; ++b) { + memcpy(forward_inputs + b * current_len, sequence_buffer + b * longest_seq, current_len * sizeof(int)); + } + + double step_time = benchmark_gpt2_forward(&model, forward_inputs, B, current_len); + baseline_latencies.push_back(step_time); + } + + + // ================= KV CACHE RUN ================= + printf("\n=== Running KV Cache ===\n"); + g_enable_kv_cache = true; + if (g_kv_cache) { cudaFree(g_kv_cache); g_kv_cache = nullptr; } + g_is_prefill = true; + g_layer_idx = 0; + current_len = prompt_len; + + float* kv_logits = (float*)mallocCheck(B * Vp * sizeof(float)); + std::vector kv_latencies; + int* kv_gen_tokens = (int*)mallocCheck(B * gen_len * sizeof(int)); // track KV-sampled tokens + + for (int b = 0; b < B; ++b) { + memcpy(forward_inputs + b * current_len, sequence_buffer + b * longest_seq, current_len * sizeof(int)); + } + + // Prefill KV + double kv_prefill_time = benchmark_gpt2_forward(&model, forward_inputs, B, current_len); + printf("KV Prefill latency: %.3f ms\n", kv_prefill_time); + + int decode_input[1024]; // max batch size + + bool ok = true; + for (int t = 0; t < gen_len; ++t) { + float* ref_logits = baseline_logits + t * B * Vp; + + // After prefill (t==0), output has shape (B, prompt_len, Vp) — extract last token. + // After decode steps (t>0), output has shape (B, 1, Vp) — just copy directly. + if (t == 0) { + copy_last_logits(kv_logits, model.acts.output, B, current_len, Vp); + } else { + copy_single_token_logits(kv_logits, model.acts.output, B, Vp); + } + + // ACCURACY CHECK + double step_maxerr = 0; + double rmse = 0; + for (int b = 0; b < B; ++b) { + for (int v = 0; v < V; ++v) { + double expected = ref_logits[b * Vp + v]; + double actual = kv_logits[b * Vp + v]; + double abserr = abs(actual - expected); + double relerr = abs(actual - expected) / fmaxf(1e-3, abs(expected)); + double err = fmin(abserr, relerr); + step_maxerr = fmax(step_maxerr, err); + rmse += err * err; + + if (err > 0.1) { + printf("=========================MISMATCH AT TOKEN %d BATCH %d VOCAB %d: ", t, b, v); + printf("%f %f (err=%f)=========================\n", expected, actual, err); + ok = false; + break; + } + } + if (!ok) break; + } + if (!ok) break; + + // Record what KV cache would have sampled (for debugging output) + for (int b = 0; b < B; ++b) { + kv_gen_tokens[t * B + b] = sample_token_argmax(kv_logits + b * Vp, V); + } + + // Use the BASELINE's sampled token for both paths to prevent sequence + // divergence from tiny numerical differences that pass the error tolerance. + // sequence_buffer already contains the baseline-sampled tokens from the baseline run. + for (int b = 0; b < B; ++b) { + decode_input[b] = sequence_buffer[b * longest_seq + current_len]; + } + + g_current_pos = current_len; + current_len++; + + // KV Step + double step_time = benchmark_gpt2_forward(&model, decode_input, B, 1); + kv_latencies.push_back(step_time); + } + + if (ok) { + printf("\n=================== KV CACHE PASSED ALL ACCURACY TESTS ===================\n"); + printf("Speedup Analysis:\n"); + double avg_baseline = 0; + double avg_kv = 0; + for (int t = 0; t < gen_len; ++t) { + double b_ms = baseline_latencies[t]; + double k_ms = kv_latencies[t]; + avg_baseline += b_ms; + avg_kv += k_ms; + if (t < 5 || t == gen_len - 1) { + printf("Token %3d: Baseline = %7.3f ms | KV Cache = %7.3f ms | Speedup = %5.2fx\n", + t, b_ms, k_ms, b_ms / k_ms); + } + if (t == 5 && gen_len > 6) printf("...\n"); + } + avg_baseline /= gen_len; + avg_kv /= gen_len; + printf("---------------------------------------------------------------------------\n"); + printf("AVERAGE: Baseline = %7.3f ms | KV Cache = %7.3f ms | Speedup = %5.2fx\n", + avg_baseline, avg_kv, avg_baseline / avg_kv); + + // Print generated text for deterministic prompts + if (!args.random_mode && tokenizer.init_ok) { + printf("\nGenerated Text:\n"); + for (int b = 0; b < B; ++b) { + printf("--- Batch %d ---\n", b); + printf("[Prompt] "); + for (int t = 0; t < prompt_len; ++t) { + safe_printf(tokenizer_decode(&tokenizer, sequence_buffer[b * longest_seq + t])); + } + printf("\n[Baseline] "); + for (int t = 0; t < gen_len; ++t) { + safe_printf(tokenizer_decode(&tokenizer, sequence_buffer[b * longest_seq + prompt_len + t])); + } + printf("\n[KV Cache] "); + for (int t = 0; t < gen_len; ++t) { + safe_printf(tokenizer_decode(&tokenizer, kv_gen_tokens[t * B + b])); + } + printf("\n\n"); + } + } + } + + free(sequence_buffer); + free(forward_inputs); + free(baseline_logits); + free(kv_logits); + free(kv_gen_tokens); + if (g_kv_cache) cudaFree(g_kv_cache); + gpt2_free(&model); + tokenizer_free(&tokenizer); + + return ok ? 0 : 1; +} diff --git a/verify_kv_cache.slurm b/verify_kv_cache.slurm new file mode 100644 index 0000000..70f07b3 --- /dev/null +++ b/verify_kv_cache.slurm @@ -0,0 +1,45 @@ +#!/bin/bash +#SBATCH --job-name="verify_kv_cache" +#SBATCH --output="verify_kv_cache_output.out" +#SBATCH --error="verify_kv_cache_errors.err" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=24G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:25:00 + +module reset + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +# Deterministic Test +echo -e "================ DETERMINISTIC VERIFICATION ================\n" +srun ./verify_kv_cache + +# Random Performance Grid +echo -e "\n\n================ RANDOM PERFORMANCE GRID ================\n" + +srun ./verify_kv_cache --random --batch 80 --prompt-len 32 --gen-len 32 +srun ./verify_kv_cache --random --batch 80 --prompt-len 32 --gen-len 64 +srun ./verify_kv_cache --random --batch 80 --prompt-len 32 --gen-len 128 + +# srun ./verify_kv_cache --random --batch 80 --prompt-len 64 --gen-len 32 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 64 --gen-len 64 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 64 --gen-len 128 + +# srun ./verify_kv_cache --random --batch 80 --prompt-len 128 --gen-len 32 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 128 --gen-len 64 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 128 --gen-len 128 + +# srun ./verify_kv_cache --random --batch 80 --prompt-len 256 --gen-len 32 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 256 --gen-len 64 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 256 --gen-len 128 + +# srun ./verify_kv_cache --random --batch 80 --prompt-len 512 --gen-len 32 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 512 --gen-len 64 +# srun ./verify_kv_cache --random --batch 80 --prompt-len 512 --gen-len 128 diff --git a/verify_ncu.slurm b/verify_ncu.slurm new file mode 100644 index 0000000..896612c --- /dev/null +++ b/verify_ncu.slurm @@ -0,0 +1,47 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects,perf,nvperf" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:15:00 +#SBATCH --exclude=gpub029,gpub034,gpub038,gpub043,gpub074,gpub054,gpub012 + +module reset +module load cuda + +SUFFIX=${1:-op9} +BIN=./build/test_gpt2_${SUFFIX} + +# ------------------------- +# output dirs +# ------------------------- +OUTDIR=output/perf/ncu +REPORTDIR=output/perf/reports + +mkdir -p $OUTDIR +mkdir -p $REPORTDIR + +OUTFILE=$OUTDIR/verify_${SUFFIX}.out +ERRFILE=$OUTDIR/verify_${SUFFIX}.err + +REPORT=$REPORTDIR/analysis_${SUFFIX} + +echo "Binary: $BIN" +echo "NCU report: $REPORT" + +exec > $OUTFILE 2> $ERRFILE + +echo -e "job $SLURM_JOBID starting on $(hostname)\n\n" + +# ------------------------- +# NCU profiling +# ------------------------- +srun ncu --set full -f \ + -o $REPORT \ + $BIN \ No newline at end of file diff --git a/verify_new.slurm b/verify_new.slurm new file mode 100644 index 0000000..f9f7102 --- /dev/null +++ b/verify_new.slurm @@ -0,0 +1,28 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --output="verify_output.out" +#SBATCH --error="verify_errors.err" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects,perf,nvperf" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:15:00 +#SBATCH --exclude=gpub029,gpub034,gpub038,gpub043,gpub074,gpub054,gpub012 +#SBATCH --mail-user=asing271@illinois.edu +#SBATCH --mail-type=BEGIN,END,FAIL + +module reset +module load cuda + +echo -e "job $SLURM_JOBID is starting on `hostname`\n\n" + +# ── nsys system-level profile ───────────────────────────────────── +srun nsys profile --stats=true ./test_gpt2 > test_gpt2_nsys_prof_req_0.out + +# ── ncu kernel-level profile ────────────────────────────────────── +# ncu --set full -f -o analysis_file ./test_gpt2 > test_gpt2_ncu.out \ No newline at end of file diff --git a/verify_nsys.slurm b/verify_nsys.slurm new file mode 100644 index 0000000..ac5165b --- /dev/null +++ b/verify_nsys.slurm @@ -0,0 +1,49 @@ +#!/bin/bash +#SBATCH --job-name="verify" +#SBATCH --partition=gpuA40x4 +#SBATCH --mem=12G +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --cpus-per-task=1 +#SBATCH --constraint="projects,perf,nvperf" +#SBATCH --gpus-per-node=1 +#SBATCH --gpu-bind=closest +#SBATCH --account=bche-delta-gpu +#SBATCH -t 00:15:00 +#SBATCH --exclude=gpub029,gpub034,gpub038,gpub043,gpub074,gpub054,gpub012 + +module reset +module load cuda + +SUFFIX=${1:-op9} +BIN=./build/test_gpt2_${SUFFIX} + +# ------------------------- +# output dirs +# ------------------------- +OUTDIR=output/perf/nsys +REPORTDIR=output/perf/reports + +mkdir -p $OUTDIR +mkdir -p $REPORTDIR + +OUTFILE=$OUTDIR/verify_${SUFFIX}.out +ERRFILE=$OUTDIR/verify_${SUFFIX}.err + +REPORT=$REPORTDIR/report_${SUFFIX} + +echo "Binary: $BIN" +echo "NSYS report: $REPORT" + +# redirect only SLURM logs +exec > $OUTFILE 2> $ERRFILE + +echo -e "job $SLURM_JOBID starting on $(hostname)\n\n" + +# ------------------------- +# NSYS profiling +# ------------------------- +srun nsys profile \ + --stats=true \ + --output=$REPORT \ + $BIN \ No newline at end of file