TODO for why use_fast=False is giving inconsistent behaviour (no spaces decoding invididual tokens) for Mistral-Large-Instruct-2407-4bit

710e5a31e757c619ed4165619a7f6439df67ee1c · 2024-08-22 14:45:08 +0100 · Alex Cheema

Files touched

M test/test_mistral_tokenizer.py

Diff

commit 710e5a31e757c619ed4165619a7f6439df67ee1c
Author: Alex Cheema <alexcheema123@gmail.com>
Date:   Thu Aug 22 14:45:08 2024 +0100

    TODO for why use_fast=False is giving inconsistent behaviour (no spaces decoding invididual tokens) for Mistral-Large-Instruct-2407-4bit
---
 test/test_mistral_tokenizer.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/test_mistral_tokenizer.py b/test/test_mistral_tokenizer.py
index 0d6f57fd..1f986796 100644
--- a/test/test_mistral_tokenizer.py
+++ b/test/test_mistral_tokenizer.py
@@ -39,6 +39,7 @@ models = [
 import os
 verbose = os.environ.get("VERBOSE", "0").lower() == "1"
 for m in models:
+    # TODO: figure out why use_fast=False is giving inconsistent behaviour (no spaces decoding invididual tokens) for Mistral-Large-Instruct-2407-4bit
     # test_tokenizer(m, AutoProcessor.from_pretrained(m, use_fast=False), verbose)
     test_tokenizer(m, AutoProcessor.from_pretrained(m, use_fast=True), verbose)
     test_tokenizer(m, AutoTokenizer.from_pretrained(m), verbose)

← e17e5f9a tests for tokenizers. unfortunately use_fast=False and use_f · back to Exo · run on every commit on main, reuqire approval on other branc ad695696 →