Finetune crash with Mistral Model #3578

Drael64 · 2023-10-11T08:44:26Z

Ran with the following parameters with a fresh llama.cpp build, built with w64devkit (windows, 32gb ram, 6600M). Edited out the model names. Maybe I'm silly and am doing something wrong? 🤷‍♂️

./finetune --model-base SynthIA-7B-v1.3-Q8_0.gguf --checkpoint-out chk-lora-SynthIA-7B-v1.3-Q8_0-prose-ITERATION.gguf --lora-out lora-SynthIA-7B-v1.3-Q8_0-prose-ITERATION.bin --train-data "prose.txt" --save-every 10 --threads 6 --adam-iter 30 --batch 4 --ctx 256

Text from crash:
`main: seed: 1697013706
main: model base = 'SynthIA-7B-v1. llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - kv 19: llama_model_loader: - type f32: llama_model_loader: - type q8_0: llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: f_clamp_kqv llm_load_print_meta: f_max_alibi_bias llm_load_print_meta: n_ff llm_load_print_meta: freq_base_train llm_load_print_meta: freq_scale_train llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: mem required .................................. llama_new_context_with_model: n_ctx llama_new_context_with_model: freq_base llama_new_context_with_model: llama_new_context_with_model: kv self size = llama_new_context_with_model: main: init model
GGML_ASSERT: ggml-alloc.c:116: 3-Q8_0.gguf'
meta data with 20 key-value pairs and 291 tensors from SynthIA-7B-v1.3-Q8_0.gguf (version GGUF V2 (latest))
token_embd.weight q8_0 [ 4096, 32000, 1, 1 ]
blk.0.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.0.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.0.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.0.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.0.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.0.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.0.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.1.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.1.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.1.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.1.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.1.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.2.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.2.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.2.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.2.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.2.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.3.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.3.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.3.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.3.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.3.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.4.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.4.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.4.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.4.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.4.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.5.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.5.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.5.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.5.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.5.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.6.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.6.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.6.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.6.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.6.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.7.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.7.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.7.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.7.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.7.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.8.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.8.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.8.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.8.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.8.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.9.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.9.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.9.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.9.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.9.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.10.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.10.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.10.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.10.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.10.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.11.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.11.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.11.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.11.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.11.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.12.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.12.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.12.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.12.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.12.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.13.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.13.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.13.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.13.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.13.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.14.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.14.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.14.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.14.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.14.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.15.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.15.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.15.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.15.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.15.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.16.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.16.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.16.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.16.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.16.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.17.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.17.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.17.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.17.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.17.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.18.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.18.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.18.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.18.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.18.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.19.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.19.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.19.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.19.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.19.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.20.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.20.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.20.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.20.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.20.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.21.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.21.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.21.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.21.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.21.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.22.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.22.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.22.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.22.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.22.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.23.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.23.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.23.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.23.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.23.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.24.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.24.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.24.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.24.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.24.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.24.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.25.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.25.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.25.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.25.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.25.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.26.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.26.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.26.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.26.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.26.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.27.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.27.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.27.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.27.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.27.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.28.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.28.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.28.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.28.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.28.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.29.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.29.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.29.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.29.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.29.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.30.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.30.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.30.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.30.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.30.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.attn_q.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.31.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ]
blk.31.attn_output.weight q8_0 [ 4096, 4096, 1, 1 ]
blk.31.ffn_gate.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.31.ffn_up.weight q8_0 [ 4096, 14336, 1, 1 ]
blk.31.ffn_down.weight q8_0 [ 14336, 4096, 1, 1 ]
blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
output_norm.weight f32 [ 4096, 1, 1, 1 ]
output.weight q8_0 [ 4096, 32000, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
llama.rope.freq_base f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
tokenizer.ggml.unknown_token_id u32
general.quantization_version u32
65 tensors
226 tensors
= GGUF V2 (latest)
= llama
= SPM
= 32000
= 0
= 32768
= 4096
= 32
= 8
= 32
= 128
= 4
= 0.0e+00
= 1.0e-05
= 0.0e+00
= 0.0e+00
= 14336
= 10000.0
= 1
= 7B
= mostly Q8_0
= 7.24 B
= 7.17 GiB (8.50 BPW)
= models
token = 1 ''
token = 2 ''
token = 0 ''
= 13 '<0x0A>'
0.10 MB
= 7338.74 MB
.................................................................
= 512
= 10000.0
freq_scale = 1
64.00 MB
compute buffer total size = 79.13 MB
tensor->data == NULL

Fosowl · 2023-10-11T12:57:22Z

I am facing the same issue with model llama2 7B as well :

I run finetuning with :

./finetune --model-base ./llama-2-7b.Q4_0.gguf --checkpoint-in chk-lora-llama-2-7b-q8_0-shakespeare-LATEST.gguf --checkpoint-out chk-lora-llama-2-7b-q8_0-shakespeare-ITERATION.gguf --lora-out lora-llama-2-7b-q8_0-shakespeare-ITERATION.bin --train-data "./shakespeare.txt" --save-every 20 --threads 4 --adam-iter 150 --batch 4 --ctx 64 --use-checkpointing

which result in :

...
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_print_meta: format           = GGUF V2 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = mostly Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) 
llm_load_print_meta: general.name   = LLaMA v2
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.10 MB
llm_load_tensors: mem required  = 3647.96 MB
..................................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size  =  256.00 MB
llama_new_context_with_model: compute buffer total size = 76.63 MB
main: init model
GGML_ASSERT: /root/llama.cpp/ggml-alloc.c:116: tensor->data == NULL
Aborted (core dumped)

I use shakespear.txt as a dataset for testing : https://raw.githubusercontent.com/brunoklein99/deep-learning-notes/master/shakespeare.txt

I use Q4_0 model from the bloke : https://huggingface.co/TheBloke/Llama-2-7B-GGUF/tree/main

Tried on mac AMD rx 570 and nvidia RTX A4000 rented on vast.ai and encounter the same error, does anyone know a workaround ?

Fosowl · 2023-10-11T13:25:04Z

commit a03ce38 seem to be working

Drael64 · 2023-10-11T13:25:25Z

Yeah, I just tried this on Xwin, which is a llama-2 model, and got the same error. I'm also running an AMD cpu/gpu if that makes any difference

Drael64 · 2023-10-11T13:27:07Z

commit a03ce38 seem to be working

I believe that's the code I'm running. I'll check, but pretty sure. Yup, that's the code, with lines 335, and 336:
assert_shape_2d(layer.wk, hparams.n_embd, hparams.n_embd_gqa());
assert_shape_2d(layer.wv, hparams.n_embd, hparams.n_embd_gqa());

slaren · 2023-10-11T13:56:38Z

This probably caused because #3548 enabled this assert all the time, not just on ggml-alloc debug builds. I am not sure what is the finetune example doing to hit these asserts, but this shouldn't happen in normal cases.

For now, you can comment the GGML_ASSERT(tensor->data == NULL) in ggml-alloc.c to restore the previous behavior.

Drael64 · 2023-10-11T16:32:23Z

This probably caused because #3548 enabled this assert all the time, not just on ggml-alloc debug builds. I am not sure what is the finetune example doing to hit these asserts, but this shouldn't happen in normal cases.

For now, you can comment the GGML_ASSERT(tensor->data == NULL) in ggml-alloc.c to restore the previous behavior.

That works. I'm training now. Let's see if I'm any good at this lol. Thanks so much for your help and the work you do on this project too :)

Fosowl · 2023-10-11T16:46:32Z

thank you ! may I ask if using quantized q4_0 model work for finetuning or should I use q8 ?

mirek190 · 2023-10-12T19:28:54Z

7b mdel and q4 ... is bad combination ....

slaren mentioned this issue Oct 12, 2023

ggml : add context enumeration functions #3605

Merged

slaren closed this as completed in #3605 Oct 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finetune crash with Mistral Model #3578

Finetune crash with Mistral Model #3578

Drael64 commented Oct 11, 2023 •

edited

Loading

Fosowl commented Oct 11, 2023 •

edited

Loading

Fosowl commented Oct 11, 2023

Drael64 commented Oct 11, 2023 •

edited

Loading

Drael64 commented Oct 11, 2023

slaren commented Oct 11, 2023

Drael64 commented Oct 11, 2023

Fosowl commented Oct 11, 2023

mirek190 commented Oct 12, 2023

Finetune crash with Mistral Model #3578

Finetune crash with Mistral Model #3578

Comments

Drael64 commented Oct 11, 2023 • edited Loading

Fosowl commented Oct 11, 2023 • edited Loading

Fosowl commented Oct 11, 2023

Drael64 commented Oct 11, 2023 • edited Loading

Drael64 commented Oct 11, 2023

slaren commented Oct 11, 2023

Drael64 commented Oct 11, 2023

Fosowl commented Oct 11, 2023

mirek190 commented Oct 12, 2023

Drael64 commented Oct 11, 2023 •

edited

Loading

Fosowl commented Oct 11, 2023 •

edited

Loading

Drael64 commented Oct 11, 2023 •

edited

Loading