job 18004587 starting on gpub005.delta.ncsa.illinois.edu


==PROF== Connected to process 408869 (/u/asingh41/sp26_ece408_Temperature1.5/build/test_gpt2_op_16)
==PROF== Profiling "encoder_forward_kernel" - 0: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 1: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 2: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 3: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 4: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 5: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 6: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 7: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 8: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 9: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 10: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 11: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 12: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 13: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 14: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 15: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 16: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 17: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 18: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 19: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 20: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 21: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 22: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 23: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 24: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 25: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 26: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 27: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 28: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 29: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 30: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 31: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 32: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 33: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 34: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 35: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 36: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 37: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 38: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 39: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 40: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 41: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 42: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 43: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 44: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 45: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 46: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 47: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 48: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 49: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 50: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 51: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 52: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 53: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 54: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 55: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 56: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 57: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 58: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 59: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 60: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 61: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 62: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 63: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 64: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 65: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 66: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 67: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 68: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 69: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 70: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 71: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 72: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 73: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 74: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 75: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 76: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 77: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 78: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 79: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 80: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 81: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 82: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 83: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 84: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 85: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 86: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 87: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 88: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 89: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 90: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 91: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 92: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 93: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 94: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 95: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 96: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 97: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 98: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 99: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 100: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 101: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 102: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 103: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 104: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 105: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 106: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 107: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 108: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 109: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 110: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 111: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 112: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 113: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 114: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 115: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 116: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 117: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 118: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 119: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 120: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 121: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 122: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 123: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 124: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 125: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 126: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 127: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 128: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 129: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 130: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 131: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 132: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 133: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 134: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 135: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 136: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 137: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 138: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 139: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 140: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 141: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 142: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 143: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 144: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 145: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 146: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 147: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 148: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 149: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 150: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 151: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 152: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 153: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 154: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 155: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 156: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 157: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 158: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 159: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 160: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 161: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 162: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 163: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 164: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 165: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 166: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 167: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 168: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 169: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 170: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 171: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 172: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 173: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 174: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 175: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 176: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 177: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 178: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 179: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 180: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 181: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 182: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 183: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 184: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 185: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 186: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 187: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 188: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 189: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 190: 0%....50%....100% - 40 passes
==PROF== Profiling "permute_kernel" - 191: 0%....50%....100% - 40 passes
==PROF== Profiling "preattention_scores_kernel" - 192: 0%....50%....100% - 40 passes
==PROF== Profiling "softmax_forward_kernel" - 193: 0%....50%....100% - 40 passes
==PROF== Profiling "attention_v_kernel" - 194: 0%....50%....100% - 40 passes
==PROF== Profiling "unpermute_kernel" - 195: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x32_sliced1x4_tn" - 196: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 197: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 198: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 199: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 200: 0%....50%....100% - 40 passes
==PROF== Profiling "Kernel" - 201: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_64x64_tn" - 202: 0%....50%....100% - 40 passes
==PROF== Profiling "bias_add_filler_kernel" - 203: 0%....50%....100% - 40 passes
==PROF== Profiling "residual_forward_kernel" - 204: 0%....50%....100% - 40 passes
==PROF== Profiling "layernorm_forward_kernel" - 205: 0%....50%....100% - 40 passes
==PROF== Profiling "ampere_sgemm_128x64_tn" - 206: 0%....50%....100% - 40 passes
test-gpt2-kernels - TF32 Mode is turned OFF.
test-gpt2 - [State]
Batch Size: 4
Sequence Length: 64
GPT2 - Allocated 221 MiB for activations
test-gpt2 - Printing 10 Expected vs Actual Logits: 
-43.431618, -43.315117
-39.836346, -39.732384
-43.065910, -42.960098
-42.828045, -42.723515
-43.529541, -43.418613
-44.318398, -44.210033
-41.227425, -41.119377
-41.270760, -41.164490
-42.541393, -42.428894
-42.394997, -42.288929
---------------- Logits OK! (max err 0.082334, RMSE 0.002728) ----------------
_________________________ All tests passed! _________________________
==PROF== Disconnected from process 408869
==PROF== Report: /u/asingh41/sp26_ece408_Temperature1.5/output/perf/reports/analysis_op_16.ncu-rep