[
  {
    "objectID": "followup_curriculum_plan.html",
    "href": "followup_curriculum_plan.html",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "This follow-up sequence assumes a single consumer workstation: roughly 8-24 GB of VRAM, enough system RAM to memory-map larger quantized files, and a local stack built around llama.cpp, GGUF files, Hugging Face tools, and small PyTorch fine-tuning jobs.\n\n\nInventory the machine, the runtime, and the local model zoo before writing application code. The chapter teaches what model families, quantization formats, context windows, and memory budgets mean in practice.\nDeliverable: a hardware/model inventory table and a practical fit estimate for the GGUF files already on disk.\n\n\n\nCompare full-precision and quantized GGUF files, inspect metadata, and run small inference/latency tests. The goal is to connect quantization labels such as BF16, Q8_0, and Q4_K_M to file size, memory pressure, and throughput.\nDeliverable: a local quantization comparison and a reproducible inference timing harness.\n\n\n\nStart llama-server, query it through OpenAI-compatible HTTP endpoints, measure latency, and shut it down cleanly. This makes local models usable from normal application code.\nDeliverable: a working local API server pattern using the GGUF stack in ~/tmp/llama-cpp.\n\n\n\nReplace ad hoc prompting with typed interfaces: JSON schema, constrained decoding, retries, and validation.\nDeliverable: a small extraction API with schema checks and regression tests.\n\n\n\nBuild retrieval-augmented generation from separate pieces: chunking, embeddings, hybrid retrieval, reranking, prompt assembly, citations, and evals.\nDeliverable: a local document QA system with retrieval metrics separated from generation metrics.\n\n\n\nTrain small adapters instead of full models. Cover target modules, rank, alpha, dropout, 4-bit loading, adapter merging, and export formats.\nDeliverable: a QLoRA adapter for a 1B-8B model that fits consumer hardware.\n\n\n\nUse language-model infrastructure for structured prediction rather than chat. Compare classification heads, sequence classification, pairwise ranking, and listwise discrete choice against classical baselines.\nDeliverable: a benchmark on the retail delivery choice data with conditional logit and neural-choice baselines.\n\n\n\nIntroduce DPO-style preference tuning and show when it is a better fit than supervised fine-tuning.\nDeliverable: a small preference dataset, a tuned adapter, and a pairwise evaluation.\n\n\n\nMake evaluation reproducible: golden sets, task metrics, judge prompts, pairwise comparisons, latency, memory, and cost.\nDeliverable: a single evaluation runner that compares prompted, RAG, and fine-tuned systems.\n\n\n\nTreat agents as bounded workflow systems: typed tools, state, retries, logging, permissions, and human approval.\nDeliverable: a local model driving a narrow, auditable tool workflow.\n\n\n\nRun small vision-language and embedding models locally. Cover OCR-like extraction, image search, captioning, and document understanding.\nDeliverable: a local image/text retrieval and captioning notebook.\n\n\n\nVersion models, adapters, prompts, datasets, evals, and server settings. Track licenses and quant provenance.\nDeliverable: a lightweight model-ops checklist and reproducible local deployment folder.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-13-the-local-model-stack",
    "href": "followup_curriculum_plan.html#chapter-13-the-local-model-stack",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Inventory the machine, the runtime, and the local model zoo before writing application code. The chapter teaches what model families, quantization formats, context windows, and memory budgets mean in practice.\nDeliverable: a hardware/model inventory table and a practical fit estimate for the GGUF files already on disk.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-14-quantization-and-local-inference",
    "href": "followup_curriculum_plan.html#chapter-14-quantization-and-local-inference",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Compare full-precision and quantized GGUF files, inspect metadata, and run small inference/latency tests. The goal is to connect quantization labels such as BF16, Q8_0, and Q4_K_M to file size, memory pressure, and throughput.\nDeliverable: a local quantization comparison and a reproducible inference timing harness.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-15-serving-models-locally",
    "href": "followup_curriculum_plan.html#chapter-15-serving-models-locally",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Start llama-server, query it through OpenAI-compatible HTTP endpoints, measure latency, and shut it down cleanly. This makes local models usable from normal application code.\nDeliverable: a working local API server pattern using the GGUF stack in ~/tmp/llama-cpp.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-16-modern-prompt-interfaces",
    "href": "followup_curriculum_plan.html#chapter-16-modern-prompt-interfaces",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Replace ad hoc prompting with typed interfaces: JSON schema, constrained decoding, retries, and validation.\nDeliverable: a small extraction API with schema checks and regression tests.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-17-retrieval-that-holds-up",
    "href": "followup_curriculum_plan.html#chapter-17-retrieval-that-holds-up",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Build retrieval-augmented generation from separate pieces: chunking, embeddings, hybrid retrieval, reranking, prompt assembly, citations, and evals.\nDeliverable: a local document QA system with retrieval metrics separated from generation metrics.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-18-lora-and-qlora-fine-tuning",
    "href": "followup_curriculum_plan.html#chapter-18-lora-and-qlora-fine-tuning",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Train small adapters instead of full models. Cover target modules, rank, alpha, dropout, 4-bit loading, adapter merging, and export formats.\nDeliverable: a QLoRA adapter for a 1B-8B model that fits consumer hardware.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-19-llms-as-predictive-models",
    "href": "followup_curriculum_plan.html#chapter-19-llms-as-predictive-models",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Use language-model infrastructure for structured prediction rather than chat. Compare classification heads, sequence classification, pairwise ranking, and listwise discrete choice against classical baselines.\nDeliverable: a benchmark on the retail delivery choice data with conditional logit and neural-choice baselines.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-20-preference-optimization-on-a-budget",
    "href": "followup_curriculum_plan.html#chapter-20-preference-optimization-on-a-budget",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Introduce DPO-style preference tuning and show when it is a better fit than supervised fine-tuning.\nDeliverable: a small preference dataset, a tuned adapter, and a pairwise evaluation.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-21-evaluation-harnesses",
    "href": "followup_curriculum_plan.html#chapter-21-evaluation-harnesses",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Make evaluation reproducible: golden sets, task metrics, judge prompts, pairwise comparisons, latency, memory, and cost.\nDeliverable: a single evaluation runner that compares prompted, RAG, and fine-tuned systems.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-22-tools-and-agents",
    "href": "followup_curriculum_plan.html#chapter-22-tools-and-agents",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Treat agents as bounded workflow systems: typed tools, state, retries, logging, permissions, and human approval.\nDeliverable: a local model driving a narrow, auditable tool workflow.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-23-multimodal-local-models",
    "href": "followup_curriculum_plan.html#chapter-23-multimodal-local-models",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Run small vision-language and embedding models locally. Cover OCR-like extraction, image search, captioning, and document understanding.\nDeliverable: a local image/text retrieval and captioning notebook.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "followup_curriculum_plan.html#chapter-24-deployment-for-one-person",
    "href": "followup_curriculum_plan.html#chapter-24-deployment-for-one-person",
    "title": "Follow-up Curriculum for Consumer-Hardware LLM Work",
    "section": "",
    "text": "Version models, adapters, prompts, datasets, evals, and server settings. Track licenses and quant provenance.\nDeliverable: a lightweight model-ops checklist and reproducible local deployment folder.",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "Follow-up plan"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter15/Chapter 15 - Serving Models Locally.html",
    "href": "Hands-On-Large-Language-Models/chapter15/Chapter 15 - Serving Models Locally.html",
    "title": "Chapter 15 - Serving Models Locally",
    "section": "",
    "text": "This chapter wraps a GGUF model in a local HTTP service. The pattern is deliberately plain: start llama-server, wait until /v1/models responds, send requests, collect timings, and terminate the process.\n\nfrom __future__ import annotations\n\nimport contextlib\nimport shlex\nimport socket\nimport subprocess\nimport time\nfrom pathlib import Path\n\nimport pandas as pd\nimport requests\n\npd.set_option(\"display.max_colwidth\", 120)\n\nLLAMA_ROOT = Path.home() / \"tmp\" / \"llama-cpp\"\nSERVER = LLAMA_ROOT / \"llama.cpp\" / \"build\" / \"bin\" / \"llama-server\"\nMODEL = LLAMA_ROOT / \"models\" / \"Qwen_Qwen3.5-4B-Q4_K_M.gguf\"\n\nFIXED_FLAGS = [\n    (\"-m\", str(MODEL)),\n    (\"-c\", \"1024\"),\n    (\"-np\", \"1\"),\n    (\"--cache-type-k\", \"q8_0\"),\n    (\"--cache-type-v\", \"q8_0\"),\n    (\"--alias\", \"qwen-local\"),\n    (\"--host\", \"127.0.0.1\"),\n]\nSERVER_ATTEMPTS = [\n    {\"mode\": \"full_gpu\", \"gpu_layers\": \"all\", \"flash_attn\": \"on\"},\n    {\"mode\": \"cpu_only\", \"gpu_layers\": \"0\", \"flash_attn\": \"on\"},\n]\n\nflag_table = pd.concat([\n    pd.DataFrame(FIXED_FLAGS, columns=[\"flag\", \"value\"]),\n    pd.DataFrame(SERVER_ATTEMPTS),\n], ignore_index=True)\nflag_table\n\n\n\n\n\n\n\n\nflag\nvalue\nmode\ngpu_layers\nflash_attn\n\n\n\n\n0\n-m\n/home/alal/tmp/llama-cpp/models/Qwen_Qwen3.5-4B-Q4_K_M.gguf\nNaN\nNaN\nNaN\n\n\n1\n-c\n1024\nNaN\nNaN\nNaN\n\n\n2\n-np\n1\nNaN\nNaN\nNaN\n\n\n3\n--cache-type-k\nq8_0\nNaN\nNaN\nNaN\n\n\n4\n--cache-type-v\nq8_0\nNaN\nNaN\nNaN\n\n\n5\n--alias\nqwen-local\nNaN\nNaN\nNaN\n\n\n6\n--host\n127.0.0.1\nNaN\nNaN\nNaN\n\n\n7\nNaN\nNaN\nfull_gpu\nall\non\n\n\n8\nNaN\nNaN\ncpu_only\n0\non\n\n\n\n\n\n\n\n\ndef free_port() -> int:\n    with socket.socket() as sock:\n        sock.bind((\"127.0.0.1\", 0))\n        return sock.getsockname()[1]\n\ndef server_command(port: int, gpu_layers: str = \"all\", flash_attn: str = \"on\") -> list[str]:\n    command = [str(SERVER)]\n    for flag, value in FIXED_FLAGS:\n        command.extend([flag, value])\n    command.extend([\n        \"-ngl\", gpu_layers,\n        \"-fa\", flash_attn,\n        \"--port\", str(port),\n        \"--jinja\",\n        \"--no-webui\",\n        \"--log-disable\",\n    ])\n    return command\n\n@contextlib.contextmanager\ndef running_server():\n    errors = []\n    process = None\n    try:\n        for attempt in SERVER_ATTEMPTS:\n            port = free_port()\n            command = server_command(port, attempt[\"gpu_layers\"], attempt[\"flash_attn\"])\n            process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, text=True)\n            base_url = f\"http://127.0.0.1:{port}\"\n            deadline = time.time() + 90\n            ready = False\n            while time.time() < deadline:\n                if process.poll() is not None:\n                    errors.append(f\"{attempt['mode']} exited with code {process.returncode}\")\n                    break\n                try:\n                    ready_response = requests.get(f\"{base_url}/v1/models\", timeout=2)\n                    if ready_response.ok:\n                        ready = True\n                        break\n                except requests.RequestException:\n                    time.sleep(1)\n            if ready:\n                print(f\"server_mode: {attempt['mode']}\")\n                yield base_url, command\n                return\n            if process.poll() is None:\n                process.terminate()\n                try:\n                    process.wait(timeout=15)\n                except subprocess.TimeoutExpired:\n                    process.kill()\n                    process.wait(timeout=15)\n            process = None\n        raise RuntimeError(\"server did not become ready; \" + \"; \".join(errors))\n    finally:\n        if process is not None and process.poll() is None:\n            process.terminate()\n            try:\n                process.wait(timeout=15)\n            except subprocess.TimeoutExpired:\n                process.kill()\n                process.wait(timeout=15)\n\nexample_command = server_command(8080)\nprint(shlex.join(example_command))\n\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-server -m /home/alal/tmp/llama-cpp/models/Qwen_Qwen3.5-4B-Q4_K_M.gguf -c 1024 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --alias qwen-local --host 127.0.0.1 -ngl all -fa on --port 8080 --jinja --no-webui --log-disable\n\n\n\nwith running_server() as (base_url, command):\n    models = requests.get(f\"{base_url}/v1/models\", timeout=10).json()\n    row = models[\"data\"][0]\n    print(f\"Serving from {base_url}\")\n    print(f\"Model id: {row.get('id')}\")\n    print(f\"Object type: {row.get('object')}\")\n    print(f\"Command uses {len(command)} command-line arguments.\")\n\nserver_mode: cpu_only\nServing from http://127.0.0.1:45717\nModel id: qwen-local\nObject type: model\nCommand uses 24 command-line arguments.\n\n\n\ndef native_completion(base_url: str, question: str) -> dict:\n    prompt = f\"Q: {question}\\nA:\"\n    response = requests.post(\n        f\"{base_url}/completion\",\n        json={\n            \"prompt\": prompt,\n            \"n_predict\": 72,\n            \"temperature\": 0,\n            \"stop\": [\"\\nQ:\"],\n        },\n        timeout=90,\n    )\n    response.raise_for_status()\n    payload = response.json()\n    timings = payload.get(\"timings\", {})\n    return {\n        \"question\": question,\n        \"answer\": payload.get(\"content\", \"\").strip(),\n        \"predicted_tokens\": timings.get(\"predicted_n\"),\n        \"predicted_tokens_per_s\": timings.get(\"predicted_per_second\"),\n    }\n\nquestions = [\n    \"What is the role of llama-server in a local LLM application?\",\n    \"Why bind a development server to 127.0.0.1?\",\n    \"What should an application log for local model requests?\",\n]\n\nwith running_server() as (base_url, _):\n    rows = [native_completion(base_url, question) for question in questions]\n\nserved = pd.DataFrame(rows)\nserved[\"predicted_tokens_per_s\"] = served[\"predicted_tokens_per_s\"].round(2)\nserved\n\nserver_mode: full_gpu\n\n\n\n\n\n\n\n\n\nquestion\nanswer\npredicted_tokens\npredicted_tokens_per_s\n\n\n\n\n0\nWhat is the role of llama-server in a local LLM application?\nllama-server is a tool that provides a server-side interface for running LLMs locally. It allows users to interact w...\n46\n138.30\n\n\n1\nWhy bind a development server to 127.0.0.1?\nBecause it's the only way to access the server from the local machine.\n18\n140.88\n\n\n2\nWhat should an application log for local model requests?\nAn application should log the following information for local model requests:\\n1. Timestamp\\n2. Model name\\n3. Reque...\n72\n138.31\n\n\n\n\n\n\n\n\nsummary = served[[\"predicted_tokens\", \"predicted_tokens_per_s\"]].describe().loc[[\"mean\", \"min\", \"max\"]]\nsummary\n\n\n\n\n\n\n\n\npredicted_tokens\npredicted_tokens_per_s\n\n\n\n\nmean\n45.333333\n139.163333\n\n\nmin\n18.000000\n138.300000\n\n\nmax\n72.000000\n140.880000\n\n\n\n\n\n\n\nFor application code, the important boundary is the HTTP contract, not the model binary. Keep the server command, model path, context size, quantization, and timing fields with the run metadata so later evaluations can distinguish model quality from serving configuration.\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "15. Serving models locally"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter13/Chapter 13 - The Local Model Stack.html",
    "href": "Hands-On-Large-Language-Models/chapter13/Chapter 13 - The Local Model Stack.html",
    "title": "Chapter 13 - The Local Model Stack",
    "section": "",
    "text": "This chapter inventories the local hardware, llama.cpp runtime, and GGUF files before any modeling work. The point is to make fit decisions from observed files and machine limits rather than from model names alone.\n\nfrom __future__ import annotations\n\nimport os\nimport re\nimport shutil\nimport subprocess\nfrom pathlib import Path\n\nimport pandas as pd\n\npd.set_option(\"display.max_colwidth\", 120)\n\nLLAMA_ROOT = Path.home() / \"tmp\" / \"llama-cpp\"\nLLAMA_CPP = LLAMA_ROOT / \"llama.cpp\"\nBIN_DIR = LLAMA_CPP / \"build\" / \"bin\"\nMODEL_DIR = LLAMA_ROOT / \"models\"\nLLAMA_SERVER = BIN_DIR / \"llama-server\"\nLLAMA_CLI = BIN_DIR / \"llama-cli\"\nLLAMA_BENCH = BIN_DIR / \"llama-bench\"\n\nfor name, path in {\n    \"llama.cpp root\": LLAMA_CPP,\n    \"binary directory\": BIN_DIR,\n    \"model directory\": MODEL_DIR,\n    \"llama-server\": LLAMA_SERVER,\n}.items():\n    print(f\"{name}: {path} ({'found' if path.exists() else 'missing'})\")\n\nllama.cpp root: /home/alal/tmp/llama-cpp/llama.cpp (found)\nbinary directory: /home/alal/tmp/llama-cpp/llama.cpp/build/bin (found)\nmodel directory: /home/alal/tmp/llama-cpp/models (found)\nllama-server: /home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-server (found)\n\n\n\ndef run_command(command: list[str], timeout: int = 15) -> str:\n    result = subprocess.run(command, capture_output=True, text=True, timeout=timeout)\n    text = (result.stdout or result.stderr).strip()\n    if result.returncode != 0:\n        return f\"command failed with code {result.returncode}: {text}\"\n    return text\n\nprint(\"GPU\")\nprint(run_command([\n    \"nvidia-smi\",\n    \"--query-gpu=name,memory.total,driver_version\",\n    \"--format=csv,noheader,nounits\",\n]))\n\nprint(\"\\nSystem memory\")\nprint(run_command([\"free\", \"-h\"]))\n\nprint(\"\\nllama-server build\")\nprint(run_command([str(LLAMA_SERVER), \"--version\"]))\n\nGPU\nNVIDIA GeForce RTX 5070, 12227, 580.173.02\n\nSystem memory\ntotal        used        free      shared  buff/cache   available\nMem:            31Gi        14Gi       355Mi       2.4Gi        19Gi        16Gi\nSwap:          511Mi       511Mi       172Ki\n\nllama-server build\nversion: 1458 (720d7fa40)\nbuilt with GNU 13.3.0 for Linux x86_64\n\n\n\ndef quant_label(path: Path) -> str:\n    name = path.name.upper()\n    patterns = [\"Q2_K\", \"Q3_K\", \"Q4_K_M\", \"Q4_K_S\", \"Q5_K_M\", \"Q5_K_S\", \"Q6_K\", \"Q8_0\", \"BF16\", \"F16\", \"MXFP4\"]\n    for pattern in patterns:\n        if pattern in name:\n            return pattern\n    if path.stat().st_size < 100_000_000:\n        return \"adapter/control-vector\"\n    return \"unknown\"\n\ndef size_gib(path: Path) -> float:\n    return path.stat().st_size / 1024**3\n\nmodels = sorted(MODEL_DIR.glob(\"*.gguf\"), key=lambda path: path.stat().st_size, reverse=True)\nmodel_inventory = pd.DataFrame([\n    {\n        \"file\": path.name,\n        \"quant_or_format\": quant_label(path),\n        \"size_gib\": round(size_gib(path), 2),\n        \"path\": str(path),\n    }\n    for path in models\n])\nmodel_inventory\n\n\n\n\n\n\n\n\nfile\nquant_or_format\nsize_gib\npath\n\n\n\n\n0\ngpt-oss-20b-mxfp4.gguf\nMXFP4\n11.28\n/home/alal/tmp/llama-cpp/models/gpt-oss-20b-mxfp4.gguf\n\n\n1\ntalkie-1930-13b-it-Q4_K_M.gguf\nQ4_K_M\n7.98\n/home/alal/tmp/llama-cpp/models/talkie-1930-13b-it-Q4_K_M.gguf\n\n\n2\ntalkie-Q4_K_M.gguf\nQ4_K_M\n7.87\n/home/alal/tmp/llama-cpp/models/talkie-Q4_K_M.gguf\n\n\n3\nQwen_Qwen3.5-4B-BF16.gguf\nBF16\n7.85\n/home/alal/tmp/llama-cpp/models/Qwen_Qwen3.5-4B-BF16.gguf\n\n\n4\ngemma-4-e4b-it-Q8_0.gguf\nQ8_0\n7.48\n/home/alal/tmp/llama-cpp/models/gemma-4-e4b-it-Q8_0.gguf\n\n\n5\nQwen_Qwen3.5-4B-Q4_K_M.gguf\nQ4_K_M\n2.81\n/home/alal/tmp/llama-cpp/models/Qwen_Qwen3.5-4B-Q4_K_M.gguf\n\n\n6\nqwen35_4b_steam_v1_playtime_lora-f16.gguf\nF16\n0.01\n/home/alal/tmp/llama-cpp/models/qwen35_4b_steam_v1_playtime_lora-f16.gguf\n\n\n7\nqwen35_4b_rich_history_global_layer24_cvector.gguf\nadapter/control-vector\n0.00\n/home/alal/tmp/llama-cpp/models/qwen35_4b_rich_history_global_layer24_cvector.gguf\n\n\n\n\n\n\n\n\ndef system_memory_gib() -> float:\n    meminfo = Path(\"/proc/meminfo\").read_text().splitlines()\n    total_kib = next(int(line.split()[1]) for line in meminfo if line.startswith(\"MemTotal:\"))\n    return total_kib / 1024**2\n\ndef gpu_memory_gib() -> float | None:\n    output = run_command([\n        \"nvidia-smi\",\n        \"--query-gpu=memory.total\",\n        \"--format=csv,noheader,nounits\",\n    ])\n    try:\n        return float(output.splitlines()[0].strip()) / 1024\n    except Exception:\n        return None\n\nsystem_ram_gib = system_memory_gib()\ngpu_vram_gib = gpu_memory_gib()\nusable_vram_gib = (gpu_vram_gib or 0.0) - 1.5\n\ndef fit_assessment(size: float, label: str) -> str:\n    if label == \"adapter/control-vector\":\n        return \"sidecar artifact, not a standalone model\"\n    if gpu_vram_gib and size <= usable_vram_gib - 1.0:\n        return \"full GPU offload is plausible\"\n    if size <= system_ram_gib * 0.75:\n        return \"CPU/GPU split or memory-mapped CPU inference is plausible\"\n    return \"too large for this machine without a different quantization\"\n\nfit_table = model_inventory.copy()\nfit_table[\"fit_assessment\"] = [fit_assessment(row.size_gib, row.quant_or_format) for row in fit_table.itertuples()]\nprint(f\"Observed system RAM: {system_ram_gib:.1f} GiB\")\nprint(f\"Observed GPU VRAM: {gpu_vram_gib:.1f} GiB\")\nprint(f\"Planning VRAM after overhead: {usable_vram_gib:.1f} GiB\")\nfit_table[[\"file\", \"quant_or_format\", \"size_gib\", \"fit_assessment\"]]\n\nObserved system RAM: 31.1 GiB\nObserved GPU VRAM: 11.9 GiB\nPlanning VRAM after overhead: 10.4 GiB\n\n\n\n\n\n\n\n\n\nfile\nquant_or_format\nsize_gib\nfit_assessment\n\n\n\n\n0\ngpt-oss-20b-mxfp4.gguf\nMXFP4\n11.28\nCPU/GPU split or memory-mapped CPU inference is plausible\n\n\n1\ntalkie-1930-13b-it-Q4_K_M.gguf\nQ4_K_M\n7.98\nfull GPU offload is plausible\n\n\n2\ntalkie-Q4_K_M.gguf\nQ4_K_M\n7.87\nfull GPU offload is plausible\n\n\n3\nQwen_Qwen3.5-4B-BF16.gguf\nBF16\n7.85\nfull GPU offload is plausible\n\n\n4\ngemma-4-e4b-it-Q8_0.gguf\nQ8_0\n7.48\nfull GPU offload is plausible\n\n\n5\nQwen_Qwen3.5-4B-Q4_K_M.gguf\nQ4_K_M\n2.81\nfull GPU offload is plausible\n\n\n6\nqwen35_4b_steam_v1_playtime_lora-f16.gguf\nF16\n0.01\nfull GPU offload is plausible\n\n\n7\nqwen35_4b_rich_history_global_layer24_cvector.gguf\nadapter/control-vector\n0.00\nsidecar artifact, not a standalone model\n\n\n\n\n\n\n\n\nruntime_checks = pd.DataFrame([\n    {\"component\": \"llama-server\", \"path\": str(LLAMA_SERVER), \"exists\": LLAMA_SERVER.exists(), \"executable\": os.access(LLAMA_SERVER, os.X_OK)},\n    {\"component\": \"llama-cli\", \"path\": str(LLAMA_CLI), \"exists\": LLAMA_CLI.exists(), \"executable\": os.access(LLAMA_CLI, os.X_OK)},\n    {\"component\": \"llama-bench\", \"path\": str(LLAMA_BENCH), \"exists\": LLAMA_BENCH.exists(), \"executable\": os.access(LLAMA_BENCH, os.X_OK)},\n    {\"component\": \"nvidia-smi\", \"path\": shutil.which(\"nvidia-smi\"), \"exists\": shutil.which(\"nvidia-smi\") is not None, \"executable\": shutil.which(\"nvidia-smi\") is not None},\n])\nruntime_checks\n\n\n\n\n\n\n\n\ncomponent\npath\nexists\nexecutable\n\n\n\n\n0\nllama-server\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-server\nTrue\nTrue\n\n\n1\nllama-cli\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-cli\nTrue\nTrue\n\n\n2\nllama-bench\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-bench\nTrue\nTrue\n\n\n3\nnvidia-smi\n/usr/bin/nvidia-smi\nTrue\nTrue\n\n\n\n\n\n\n\nThe practical conclusion for this workstation is that Qwen 3.5 4B Q4 is the default fast test model, Talkie 13B Q4 is plausible for inference with careful context settings, and full fine-tuning of these generation models is not the consumer-hardware target. Adapter tuning and local serving are the right next steps.\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "13. Local model stack"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "",
    "text": "from datasets import load_dataset\n\n# Prepare data and splits\ntomatoes = load_dataset(\"rotten_tomatoes\")\ntrain_data, test_data = tomatoes[\"train\"], tomatoes[\"test\"]",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#data",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#data",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "",
    "text": "from datasets import load_dataset\n\n# Prepare data and splits\ntomatoes = load_dataset(\"rotten_tomatoes\")\ntrain_data, test_data = tomatoes[\"train\"], tomatoes[\"test\"]",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#supervised-classification",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#supervised-classification",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "Supervised Classification",
    "text": "Supervised Classification\n\nHuggingFace Trainer\n\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n# Load Model and Tokenizer\nmodel_id = \"bert-base-cased\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\n\n\n\n\n\n\nSome weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\n\n\n\n\n\n\n\n\n\n\nTokenize our data.\n\nfrom transformers import DataCollatorWithPadding\n\n# Pad to the longest sequence in the batch\ndata_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n\ndef preprocess_function(examples):\n   \"\"\"Tokenize input data\"\"\"\n   return tokenizer(examples[\"text\"], truncation=True)\n\n# Tokenize train/test data\ntokenized_train = train_data.map(preprocess_function, batched=True)\ntokenized_test = test_data.map(preprocess_function, batched=True)\n\n\n\n\n\n\n\nDefine metrics.\n\nimport numpy as np\nimport evaluate\n\n\ndef compute_metrics(eval_pred):\n    \"\"\"Calculate F1 score\"\"\"\n    logits, labels = eval_pred\n    predictions = np.argmax(logits, axis=-1)\n\n    load_f1 = evaluate.load(\"f1\")\n    f1 = load_f1.compute(predictions=predictions, references=labels)[\"f1\"]\n    return {\"f1\": f1}\n\nTrain model.\n\nfrom transformers import TrainingArguments, Trainer\n\n# Training arguments for parameter tuning\ntraining_args = TrainingArguments(\n   \"model\",\n   learning_rate=2e-5,\n   per_device_train_batch_size=16,\n   per_device_eval_batch_size=16,\n   num_train_epochs=1,\n   weight_decay=0.01,\n   save_strategy=\"epoch\",\n   report_to=\"none\"\n)\n\n# Trainer which executes the training process\ntrainer = Trainer(\n   model=model,\n   args=training_args,\n   train_dataset=tokenized_train,\n   eval_dataset=tokenized_test,\n   tokenizer=tokenizer,\n   data_collator=data_collator,\n   compute_metrics=compute_metrics,\n)\n\n\ntrainer.train()\n\n\n    \n      \n      \n      [534/534 01:00, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n500\n0.424000\n\n\n\n\n\n\nTrainOutput(global_step=534, training_loss=0.4183677966228585, metrics={'train_runtime': 61.6658, 'train_samples_per_second': 138.326, 'train_steps_per_second': 8.66, 'total_flos': 227605451772240.0, 'train_loss': 0.4183677966228585, 'epoch': 1.0})\n\n\nEvaluate results.\n\ntrainer.evaluate()\n\n\n    \n      \n      \n      [67/67 00:01]\n    \n    \n\n\n\n\n\n{'eval_loss': 0.37090229988098145,\n 'eval_f1': 0.8566073102155576,\n 'eval_runtime': 3.1133,\n 'eval_samples_per_second': 342.407,\n 'eval_steps_per_second': 21.521,\n 'epoch': 1.0}\n\n\n\n\nFreeze Layers\n\n# Load Model and Tokenizer\nmodel = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nSome weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\n\n\n# Print layer names\nfor name, param in model.named_parameters():\n    print(name)\n\nbert.embeddings.word_embeddings.weight\nbert.embeddings.position_embeddings.weight\nbert.embeddings.token_type_embeddings.weight\nbert.embeddings.LayerNorm.weight\nbert.embeddings.LayerNorm.bias\nbert.encoder.layer.0.attention.self.query.weight\nbert.encoder.layer.0.attention.self.query.bias\nbert.encoder.layer.0.attention.self.key.weight\nbert.encoder.layer.0.attention.self.key.bias\nbert.encoder.layer.0.attention.self.value.weight\nbert.encoder.layer.0.attention.self.value.bias\nbert.encoder.layer.0.attention.output.dense.weight\nbert.encoder.layer.0.attention.output.dense.bias\nbert.encoder.layer.0.attention.output.LayerNorm.weight\nbert.encoder.layer.0.attention.output.LayerNorm.bias\nbert.encoder.layer.0.intermediate.dense.weight\nbert.encoder.layer.0.intermediate.dense.bias\nbert.encoder.layer.0.output.dense.weight\nbert.encoder.layer.0.output.dense.bias\nbert.encoder.layer.0.output.LayerNorm.weight\nbert.encoder.layer.0.output.LayerNorm.bias\nbert.encoder.layer.1.attention.self.query.weight\nbert.encoder.layer.1.attention.self.query.bias\nbert.encoder.layer.1.attention.self.key.weight\nbert.encoder.layer.1.attention.self.key.bias\nbert.encoder.layer.1.attention.self.value.weight\nbert.encoder.layer.1.attention.self.value.bias\nbert.encoder.layer.1.attention.output.dense.weight\nbert.encoder.layer.1.attention.output.dense.bias\nbert.encoder.layer.1.attention.output.LayerNorm.weight\nbert.encoder.layer.1.attention.output.LayerNorm.bias\nbert.encoder.layer.1.intermediate.dense.weight\nbert.encoder.layer.1.intermediate.dense.bias\nbert.encoder.layer.1.output.dense.weight\nbert.encoder.layer.1.output.dense.bias\nbert.encoder.layer.1.output.LayerNorm.weight\nbert.encoder.layer.1.output.LayerNorm.bias\nbert.encoder.layer.2.attention.self.query.weight\nbert.encoder.layer.2.attention.self.query.bias\nbert.encoder.layer.2.attention.self.key.weight\nbert.encoder.layer.2.attention.self.key.bias\nbert.encoder.layer.2.attention.self.value.weight\nbert.encoder.layer.2.attention.self.value.bias\nbert.encoder.layer.2.attention.output.dense.weight\nbert.encoder.layer.2.attention.output.dense.bias\nbert.encoder.layer.2.attention.output.LayerNorm.weight\nbert.encoder.layer.2.attention.output.LayerNorm.bias\nbert.encoder.layer.2.intermediate.dense.weight\nbert.encoder.layer.2.intermediate.dense.bias\nbert.encoder.layer.2.output.dense.weight\nbert.encoder.layer.2.output.dense.bias\nbert.encoder.layer.2.output.LayerNorm.weight\nbert.encoder.layer.2.output.LayerNorm.bias\nbert.encoder.layer.3.attention.self.query.weight\nbert.encoder.layer.3.attention.self.query.bias\nbert.encoder.layer.3.attention.self.key.weight\nbert.encoder.layer.3.attention.self.key.bias\nbert.encoder.layer.3.attention.self.value.weight\nbert.encoder.layer.3.attention.self.value.bias\nbert.encoder.layer.3.attention.output.dense.weight\nbert.encoder.layer.3.attention.output.dense.bias\nbert.encoder.layer.3.attention.output.LayerNorm.weight\nbert.encoder.layer.3.attention.output.LayerNorm.bias\nbert.encoder.layer.3.intermediate.dense.weight\nbert.encoder.layer.3.intermediate.dense.bias\nbert.encoder.layer.3.output.dense.weight\nbert.encoder.layer.3.output.dense.bias\nbert.encoder.layer.3.output.LayerNorm.weight\nbert.encoder.layer.3.output.LayerNorm.bias\nbert.encoder.layer.4.attention.self.query.weight\nbert.encoder.layer.4.attention.self.query.bias\nbert.encoder.layer.4.attention.self.key.weight\nbert.encoder.layer.4.attention.self.key.bias\nbert.encoder.layer.4.attention.self.value.weight\nbert.encoder.layer.4.attention.self.value.bias\nbert.encoder.layer.4.attention.output.dense.weight\nbert.encoder.layer.4.attention.output.dense.bias\nbert.encoder.layer.4.attention.output.LayerNorm.weight\nbert.encoder.layer.4.attention.output.LayerNorm.bias\nbert.encoder.layer.4.intermediate.dense.weight\nbert.encoder.layer.4.intermediate.dense.bias\nbert.encoder.layer.4.output.dense.weight\nbert.encoder.layer.4.output.dense.bias\nbert.encoder.layer.4.output.LayerNorm.weight\nbert.encoder.layer.4.output.LayerNorm.bias\nbert.encoder.layer.5.attention.self.query.weight\nbert.encoder.layer.5.attention.self.query.bias\nbert.encoder.layer.5.attention.self.key.weight\nbert.encoder.layer.5.attention.self.key.bias\nbert.encoder.layer.5.attention.self.value.weight\nbert.encoder.layer.5.attention.self.value.bias\nbert.encoder.layer.5.attention.output.dense.weight\nbert.encoder.layer.5.attention.output.dense.bias\nbert.encoder.layer.5.attention.output.LayerNorm.weight\nbert.encoder.layer.5.attention.output.LayerNorm.bias\nbert.encoder.layer.5.intermediate.dense.weight\nbert.encoder.layer.5.intermediate.dense.bias\nbert.encoder.layer.5.output.dense.weight\nbert.encoder.layer.5.output.dense.bias\nbert.encoder.layer.5.output.LayerNorm.weight\nbert.encoder.layer.5.output.LayerNorm.bias\nbert.encoder.layer.6.attention.self.query.weight\nbert.encoder.layer.6.attention.self.query.bias\nbert.encoder.layer.6.attention.self.key.weight\nbert.encoder.layer.6.attention.self.key.bias\nbert.encoder.layer.6.attention.self.value.weight\nbert.encoder.layer.6.attention.self.value.bias\nbert.encoder.layer.6.attention.output.dense.weight\nbert.encoder.layer.6.attention.output.dense.bias\nbert.encoder.layer.6.attention.output.LayerNorm.weight\nbert.encoder.layer.6.attention.output.LayerNorm.bias\nbert.encoder.layer.6.intermediate.dense.weight\nbert.encoder.layer.6.intermediate.dense.bias\nbert.encoder.layer.6.output.dense.weight\nbert.encoder.layer.6.output.dense.bias\nbert.encoder.layer.6.output.LayerNorm.weight\nbert.encoder.layer.6.output.LayerNorm.bias\nbert.encoder.layer.7.attention.self.query.weight\nbert.encoder.layer.7.attention.self.query.bias\nbert.encoder.layer.7.attention.self.key.weight\nbert.encoder.layer.7.attention.self.key.bias\nbert.encoder.layer.7.attention.self.value.weight\nbert.encoder.layer.7.attention.self.value.bias\nbert.encoder.layer.7.attention.output.dense.weight\nbert.encoder.layer.7.attention.output.dense.bias\nbert.encoder.layer.7.attention.output.LayerNorm.weight\nbert.encoder.layer.7.attention.output.LayerNorm.bias\nbert.encoder.layer.7.intermediate.dense.weight\nbert.encoder.layer.7.intermediate.dense.bias\nbert.encoder.layer.7.output.dense.weight\nbert.encoder.layer.7.output.dense.bias\nbert.encoder.layer.7.output.LayerNorm.weight\nbert.encoder.layer.7.output.LayerNorm.bias\nbert.encoder.layer.8.attention.self.query.weight\nbert.encoder.layer.8.attention.self.query.bias\nbert.encoder.layer.8.attention.self.key.weight\nbert.encoder.layer.8.attention.self.key.bias\nbert.encoder.layer.8.attention.self.value.weight\nbert.encoder.layer.8.attention.self.value.bias\nbert.encoder.layer.8.attention.output.dense.weight\nbert.encoder.layer.8.attention.output.dense.bias\nbert.encoder.layer.8.attention.output.LayerNorm.weight\nbert.encoder.layer.8.attention.output.LayerNorm.bias\nbert.encoder.layer.8.intermediate.dense.weight\nbert.encoder.layer.8.intermediate.dense.bias\nbert.encoder.layer.8.output.dense.weight\nbert.encoder.layer.8.output.dense.bias\nbert.encoder.layer.8.output.LayerNorm.weight\nbert.encoder.layer.8.output.LayerNorm.bias\nbert.encoder.layer.9.attention.self.query.weight\nbert.encoder.layer.9.attention.self.query.bias\nbert.encoder.layer.9.attention.self.key.weight\nbert.encoder.layer.9.attention.self.key.bias\nbert.encoder.layer.9.attention.self.value.weight\nbert.encoder.layer.9.attention.self.value.bias\nbert.encoder.layer.9.attention.output.dense.weight\nbert.encoder.layer.9.attention.output.dense.bias\nbert.encoder.layer.9.attention.output.LayerNorm.weight\nbert.encoder.layer.9.attention.output.LayerNorm.bias\nbert.encoder.layer.9.intermediate.dense.weight\nbert.encoder.layer.9.intermediate.dense.bias\nbert.encoder.layer.9.output.dense.weight\nbert.encoder.layer.9.output.dense.bias\nbert.encoder.layer.9.output.LayerNorm.weight\nbert.encoder.layer.9.output.LayerNorm.bias\nbert.encoder.layer.10.attention.self.query.weight\nbert.encoder.layer.10.attention.self.query.bias\nbert.encoder.layer.10.attention.self.key.weight\nbert.encoder.layer.10.attention.self.key.bias\nbert.encoder.layer.10.attention.self.value.weight\nbert.encoder.layer.10.attention.self.value.bias\nbert.encoder.layer.10.attention.output.dense.weight\nbert.encoder.layer.10.attention.output.dense.bias\nbert.encoder.layer.10.attention.output.LayerNorm.weight\nbert.encoder.layer.10.attention.output.LayerNorm.bias\nbert.encoder.layer.10.intermediate.dense.weight\nbert.encoder.layer.10.intermediate.dense.bias\nbert.encoder.layer.10.output.dense.weight\nbert.encoder.layer.10.output.dense.bias\nbert.encoder.layer.10.output.LayerNorm.weight\nbert.encoder.layer.10.output.LayerNorm.bias\nbert.encoder.layer.11.attention.self.query.weight\nbert.encoder.layer.11.attention.self.query.bias\nbert.encoder.layer.11.attention.self.key.weight\nbert.encoder.layer.11.attention.self.key.bias\nbert.encoder.layer.11.attention.self.value.weight\nbert.encoder.layer.11.attention.self.value.bias\nbert.encoder.layer.11.attention.output.dense.weight\nbert.encoder.layer.11.attention.output.dense.bias\nbert.encoder.layer.11.attention.output.LayerNorm.weight\nbert.encoder.layer.11.attention.output.LayerNorm.bias\nbert.encoder.layer.11.intermediate.dense.weight\nbert.encoder.layer.11.intermediate.dense.bias\nbert.encoder.layer.11.output.dense.weight\nbert.encoder.layer.11.output.dense.bias\nbert.encoder.layer.11.output.LayerNorm.weight\nbert.encoder.layer.11.output.LayerNorm.bias\nbert.pooler.dense.weight\nbert.pooler.dense.bias\nclassifier.weight\nclassifier.bias\n\n\n\nfor name, param in model.named_parameters():\n\n     # Trainable classification head\n     if name.startswith(\"classifier\"):\n        param.requires_grad = True\n\n      # Freeze everything else\n     else:\n        param.requires_grad = False\n\n\n# We can check whether the model was correctly updated\nfor name, param in model.named_parameters():\n     print(f\"Parameter: {name} ----- {param.requires_grad}\")\n\nParameter: bert.embeddings.word_embeddings.weight ----- False\nParameter: bert.embeddings.position_embeddings.weight ----- False\nParameter: bert.embeddings.token_type_embeddings.weight ----- False\nParameter: bert.embeddings.LayerNorm.weight ----- False\nParameter: bert.embeddings.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.0.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.0.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.0.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.0.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.0.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.0.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.0.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.0.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.0.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.0.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.0.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.0.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.0.output.dense.weight ----- False\nParameter: bert.encoder.layer.0.output.dense.bias ----- False\nParameter: bert.encoder.layer.0.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.0.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.1.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.1.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.1.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.1.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.1.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.1.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.1.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.1.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.1.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.1.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.1.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.1.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.1.output.dense.weight ----- False\nParameter: bert.encoder.layer.1.output.dense.bias ----- False\nParameter: bert.encoder.layer.1.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.1.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.2.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.2.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.2.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.2.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.2.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.2.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.2.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.2.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.2.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.2.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.2.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.2.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.2.output.dense.weight ----- False\nParameter: bert.encoder.layer.2.output.dense.bias ----- False\nParameter: bert.encoder.layer.2.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.2.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.3.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.3.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.3.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.3.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.3.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.3.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.3.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.3.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.3.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.3.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.3.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.3.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.3.output.dense.weight ----- False\nParameter: bert.encoder.layer.3.output.dense.bias ----- False\nParameter: bert.encoder.layer.3.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.3.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.4.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.4.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.4.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.4.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.4.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.4.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.4.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.4.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.4.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.4.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.4.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.4.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.4.output.dense.weight ----- False\nParameter: bert.encoder.layer.4.output.dense.bias ----- False\nParameter: bert.encoder.layer.4.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.4.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.5.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.5.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.5.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.5.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.5.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.5.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.5.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.5.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.5.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.5.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.5.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.5.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.5.output.dense.weight ----- False\nParameter: bert.encoder.layer.5.output.dense.bias ----- False\nParameter: bert.encoder.layer.5.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.5.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.6.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.6.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.6.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.6.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.6.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.6.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.6.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.6.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.6.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.6.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.6.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.6.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.6.output.dense.weight ----- False\nParameter: bert.encoder.layer.6.output.dense.bias ----- False\nParameter: bert.encoder.layer.6.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.6.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.7.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.7.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.7.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.7.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.7.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.7.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.7.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.7.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.7.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.7.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.7.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.7.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.7.output.dense.weight ----- False\nParameter: bert.encoder.layer.7.output.dense.bias ----- False\nParameter: bert.encoder.layer.7.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.7.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.8.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.8.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.8.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.8.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.8.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.8.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.8.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.8.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.8.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.8.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.8.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.8.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.8.output.dense.weight ----- False\nParameter: bert.encoder.layer.8.output.dense.bias ----- False\nParameter: bert.encoder.layer.8.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.8.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.9.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.9.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.9.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.9.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.9.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.9.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.9.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.9.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.9.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.9.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.9.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.9.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.9.output.dense.weight ----- False\nParameter: bert.encoder.layer.9.output.dense.bias ----- False\nParameter: bert.encoder.layer.9.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.9.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.10.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.10.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.10.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.10.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.10.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.10.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.10.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.10.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.10.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.10.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.10.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.10.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.10.output.dense.weight ----- False\nParameter: bert.encoder.layer.10.output.dense.bias ----- False\nParameter: bert.encoder.layer.10.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.10.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.11.attention.self.query.weight ----- False\nParameter: bert.encoder.layer.11.attention.self.query.bias ----- False\nParameter: bert.encoder.layer.11.attention.self.key.weight ----- False\nParameter: bert.encoder.layer.11.attention.self.key.bias ----- False\nParameter: bert.encoder.layer.11.attention.self.value.weight ----- False\nParameter: bert.encoder.layer.11.attention.self.value.bias ----- False\nParameter: bert.encoder.layer.11.attention.output.dense.weight ----- False\nParameter: bert.encoder.layer.11.attention.output.dense.bias ----- False\nParameter: bert.encoder.layer.11.attention.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.11.attention.output.LayerNorm.bias ----- False\nParameter: bert.encoder.layer.11.intermediate.dense.weight ----- False\nParameter: bert.encoder.layer.11.intermediate.dense.bias ----- False\nParameter: bert.encoder.layer.11.output.dense.weight ----- False\nParameter: bert.encoder.layer.11.output.dense.bias ----- False\nParameter: bert.encoder.layer.11.output.LayerNorm.weight ----- False\nParameter: bert.encoder.layer.11.output.LayerNorm.bias ----- False\nParameter: bert.pooler.dense.weight ----- False\nParameter: bert.pooler.dense.bias ----- False\nParameter: classifier.weight ----- True\nParameter: classifier.bias ----- True\n\n\n\nfrom transformers import TrainingArguments, Trainer\n\n# Trainer which executes the training process\ntrainer = Trainer(\n   model=model,\n   args=training_args,\n   train_dataset=tokenized_train,\n   eval_dataset=tokenized_test,\n   tokenizer=tokenizer,\n   data_collator=data_collator,\n   compute_metrics=compute_metrics,\n)\ntrainer.train()\n\n\n    \n      \n      \n      [534/534 00:15, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n500\n0.697000\n\n\n\n\n\n\nTrainOutput(global_step=534, training_loss=0.6962381677234664, metrics={'train_runtime': 15.234, 'train_samples_per_second': 559.931, 'train_steps_per_second': 35.053, 'total_flos': 227605451772240.0, 'train_loss': 0.6962381677234664, 'epoch': 1.0})\n\n\n\ntrainer.evaluate()\n\n\n    \n      \n      \n      [67/67 00:01]\n    \n    \n\n\n{'eval_loss': 0.6823198795318604,\n 'eval_f1': 0.637704918032787,\n 'eval_runtime': 2.7203,\n 'eval_samples_per_second': 391.865,\n 'eval_steps_per_second': 24.629,\n 'epoch': 1.0}\n\n\n\n\nFreeze blocks 1-5\n\n# We can check whether the model was correctly updated\nfor index, (name, param) in enumerate(model.named_parameters()):\n     print(f\"Parameter: {index}{name} ----- {param.requires_grad}\")\n\nParameter: 0bert.embeddings.word_embeddings.weight ----- False\nParameter: 1bert.embeddings.position_embeddings.weight ----- False\nParameter: 2bert.embeddings.token_type_embeddings.weight ----- False\nParameter: 3bert.embeddings.LayerNorm.weight ----- False\nParameter: 4bert.embeddings.LayerNorm.bias ----- False\nParameter: 5bert.encoder.layer.0.attention.self.query.weight ----- False\nParameter: 6bert.encoder.layer.0.attention.self.query.bias ----- False\nParameter: 7bert.encoder.layer.0.attention.self.key.weight ----- False\nParameter: 8bert.encoder.layer.0.attention.self.key.bias ----- False\nParameter: 9bert.encoder.layer.0.attention.self.value.weight ----- False\nParameter: 10bert.encoder.layer.0.attention.self.value.bias ----- False\nParameter: 11bert.encoder.layer.0.attention.output.dense.weight ----- False\nParameter: 12bert.encoder.layer.0.attention.output.dense.bias ----- False\nParameter: 13bert.encoder.layer.0.attention.output.LayerNorm.weight ----- False\nParameter: 14bert.encoder.layer.0.attention.output.LayerNorm.bias ----- False\nParameter: 15bert.encoder.layer.0.intermediate.dense.weight ----- False\nParameter: 16bert.encoder.layer.0.intermediate.dense.bias ----- False\nParameter: 17bert.encoder.layer.0.output.dense.weight ----- False\nParameter: 18bert.encoder.layer.0.output.dense.bias ----- False\nParameter: 19bert.encoder.layer.0.output.LayerNorm.weight ----- False\nParameter: 20bert.encoder.layer.0.output.LayerNorm.bias ----- False\nParameter: 21bert.encoder.layer.1.attention.self.query.weight ----- False\nParameter: 22bert.encoder.layer.1.attention.self.query.bias ----- False\nParameter: 23bert.encoder.layer.1.attention.self.key.weight ----- False\nParameter: 24bert.encoder.layer.1.attention.self.key.bias ----- False\nParameter: 25bert.encoder.layer.1.attention.self.value.weight ----- False\nParameter: 26bert.encoder.layer.1.attention.self.value.bias ----- False\nParameter: 27bert.encoder.layer.1.attention.output.dense.weight ----- False\nParameter: 28bert.encoder.layer.1.attention.output.dense.bias ----- False\nParameter: 29bert.encoder.layer.1.attention.output.LayerNorm.weight ----- False\nParameter: 30bert.encoder.layer.1.attention.output.LayerNorm.bias ----- False\nParameter: 31bert.encoder.layer.1.intermediate.dense.weight ----- False\nParameter: 32bert.encoder.layer.1.intermediate.dense.bias ----- False\nParameter: 33bert.encoder.layer.1.output.dense.weight ----- False\nParameter: 34bert.encoder.layer.1.output.dense.bias ----- False\nParameter: 35bert.encoder.layer.1.output.LayerNorm.weight ----- False\nParameter: 36bert.encoder.layer.1.output.LayerNorm.bias ----- False\nParameter: 37bert.encoder.layer.2.attention.self.query.weight ----- False\nParameter: 38bert.encoder.layer.2.attention.self.query.bias ----- False\nParameter: 39bert.encoder.layer.2.attention.self.key.weight ----- False\nParameter: 40bert.encoder.layer.2.attention.self.key.bias ----- False\nParameter: 41bert.encoder.layer.2.attention.self.value.weight ----- False\nParameter: 42bert.encoder.layer.2.attention.self.value.bias ----- False\nParameter: 43bert.encoder.layer.2.attention.output.dense.weight ----- False\nParameter: 44bert.encoder.layer.2.attention.output.dense.bias ----- False\nParameter: 45bert.encoder.layer.2.attention.output.LayerNorm.weight ----- False\nParameter: 46bert.encoder.layer.2.attention.output.LayerNorm.bias ----- False\nParameter: 47bert.encoder.layer.2.intermediate.dense.weight ----- False\nParameter: 48bert.encoder.layer.2.intermediate.dense.bias ----- False\nParameter: 49bert.encoder.layer.2.output.dense.weight ----- False\nParameter: 50bert.encoder.layer.2.output.dense.bias ----- False\nParameter: 51bert.encoder.layer.2.output.LayerNorm.weight ----- False\nParameter: 52bert.encoder.layer.2.output.LayerNorm.bias ----- False\nParameter: 53bert.encoder.layer.3.attention.self.query.weight ----- False\nParameter: 54bert.encoder.layer.3.attention.self.query.bias ----- False\nParameter: 55bert.encoder.layer.3.attention.self.key.weight ----- False\nParameter: 56bert.encoder.layer.3.attention.self.key.bias ----- False\nParameter: 57bert.encoder.layer.3.attention.self.value.weight ----- False\nParameter: 58bert.encoder.layer.3.attention.self.value.bias ----- False\nParameter: 59bert.encoder.layer.3.attention.output.dense.weight ----- False\nParameter: 60bert.encoder.layer.3.attention.output.dense.bias ----- False\nParameter: 61bert.encoder.layer.3.attention.output.LayerNorm.weight ----- False\nParameter: 62bert.encoder.layer.3.attention.output.LayerNorm.bias ----- False\nParameter: 63bert.encoder.layer.3.intermediate.dense.weight ----- False\nParameter: 64bert.encoder.layer.3.intermediate.dense.bias ----- False\nParameter: 65bert.encoder.layer.3.output.dense.weight ----- False\nParameter: 66bert.encoder.layer.3.output.dense.bias ----- False\nParameter: 67bert.encoder.layer.3.output.LayerNorm.weight ----- False\nParameter: 68bert.encoder.layer.3.output.LayerNorm.bias ----- False\nParameter: 69bert.encoder.layer.4.attention.self.query.weight ----- False\nParameter: 70bert.encoder.layer.4.attention.self.query.bias ----- False\nParameter: 71bert.encoder.layer.4.attention.self.key.weight ----- False\nParameter: 72bert.encoder.layer.4.attention.self.key.bias ----- False\nParameter: 73bert.encoder.layer.4.attention.self.value.weight ----- False\nParameter: 74bert.encoder.layer.4.attention.self.value.bias ----- False\nParameter: 75bert.encoder.layer.4.attention.output.dense.weight ----- False\nParameter: 76bert.encoder.layer.4.attention.output.dense.bias ----- False\nParameter: 77bert.encoder.layer.4.attention.output.LayerNorm.weight ----- False\nParameter: 78bert.encoder.layer.4.attention.output.LayerNorm.bias ----- False\nParameter: 79bert.encoder.layer.4.intermediate.dense.weight ----- False\nParameter: 80bert.encoder.layer.4.intermediate.dense.bias ----- False\nParameter: 81bert.encoder.layer.4.output.dense.weight ----- False\nParameter: 82bert.encoder.layer.4.output.dense.bias ----- False\nParameter: 83bert.encoder.layer.4.output.LayerNorm.weight ----- False\nParameter: 84bert.encoder.layer.4.output.LayerNorm.bias ----- False\nParameter: 85bert.encoder.layer.5.attention.self.query.weight ----- False\nParameter: 86bert.encoder.layer.5.attention.self.query.bias ----- False\nParameter: 87bert.encoder.layer.5.attention.self.key.weight ----- False\nParameter: 88bert.encoder.layer.5.attention.self.key.bias ----- False\nParameter: 89bert.encoder.layer.5.attention.self.value.weight ----- False\nParameter: 90bert.encoder.layer.5.attention.self.value.bias ----- False\nParameter: 91bert.encoder.layer.5.attention.output.dense.weight ----- False\nParameter: 92bert.encoder.layer.5.attention.output.dense.bias ----- False\nParameter: 93bert.encoder.layer.5.attention.output.LayerNorm.weight ----- False\nParameter: 94bert.encoder.layer.5.attention.output.LayerNorm.bias ----- False\nParameter: 95bert.encoder.layer.5.intermediate.dense.weight ----- False\nParameter: 96bert.encoder.layer.5.intermediate.dense.bias ----- False\nParameter: 97bert.encoder.layer.5.output.dense.weight ----- False\nParameter: 98bert.encoder.layer.5.output.dense.bias ----- False\nParameter: 99bert.encoder.layer.5.output.LayerNorm.weight ----- False\nParameter: 100bert.encoder.layer.5.output.LayerNorm.bias ----- False\nParameter: 101bert.encoder.layer.6.attention.self.query.weight ----- False\nParameter: 102bert.encoder.layer.6.attention.self.query.bias ----- False\nParameter: 103bert.encoder.layer.6.attention.self.key.weight ----- False\nParameter: 104bert.encoder.layer.6.attention.self.key.bias ----- False\nParameter: 105bert.encoder.layer.6.attention.self.value.weight ----- False\nParameter: 106bert.encoder.layer.6.attention.self.value.bias ----- False\nParameter: 107bert.encoder.layer.6.attention.output.dense.weight ----- False\nParameter: 108bert.encoder.layer.6.attention.output.dense.bias ----- False\nParameter: 109bert.encoder.layer.6.attention.output.LayerNorm.weight ----- False\nParameter: 110bert.encoder.layer.6.attention.output.LayerNorm.bias ----- False\nParameter: 111bert.encoder.layer.6.intermediate.dense.weight ----- False\nParameter: 112bert.encoder.layer.6.intermediate.dense.bias ----- False\nParameter: 113bert.encoder.layer.6.output.dense.weight ----- False\nParameter: 114bert.encoder.layer.6.output.dense.bias ----- False\nParameter: 115bert.encoder.layer.6.output.LayerNorm.weight ----- False\nParameter: 116bert.encoder.layer.6.output.LayerNorm.bias ----- False\nParameter: 117bert.encoder.layer.7.attention.self.query.weight ----- False\nParameter: 118bert.encoder.layer.7.attention.self.query.bias ----- False\nParameter: 119bert.encoder.layer.7.attention.self.key.weight ----- False\nParameter: 120bert.encoder.layer.7.attention.self.key.bias ----- False\nParameter: 121bert.encoder.layer.7.attention.self.value.weight ----- False\nParameter: 122bert.encoder.layer.7.attention.self.value.bias ----- False\nParameter: 123bert.encoder.layer.7.attention.output.dense.weight ----- False\nParameter: 124bert.encoder.layer.7.attention.output.dense.bias ----- False\nParameter: 125bert.encoder.layer.7.attention.output.LayerNorm.weight ----- False\nParameter: 126bert.encoder.layer.7.attention.output.LayerNorm.bias ----- False\nParameter: 127bert.encoder.layer.7.intermediate.dense.weight ----- False\nParameter: 128bert.encoder.layer.7.intermediate.dense.bias ----- False\nParameter: 129bert.encoder.layer.7.output.dense.weight ----- False\nParameter: 130bert.encoder.layer.7.output.dense.bias ----- False\nParameter: 131bert.encoder.layer.7.output.LayerNorm.weight ----- False\nParameter: 132bert.encoder.layer.7.output.LayerNorm.bias ----- False\nParameter: 133bert.encoder.layer.8.attention.self.query.weight ----- False\nParameter: 134bert.encoder.layer.8.attention.self.query.bias ----- False\nParameter: 135bert.encoder.layer.8.attention.self.key.weight ----- False\nParameter: 136bert.encoder.layer.8.attention.self.key.bias ----- False\nParameter: 137bert.encoder.layer.8.attention.self.value.weight ----- False\nParameter: 138bert.encoder.layer.8.attention.self.value.bias ----- False\nParameter: 139bert.encoder.layer.8.attention.output.dense.weight ----- False\nParameter: 140bert.encoder.layer.8.attention.output.dense.bias ----- False\nParameter: 141bert.encoder.layer.8.attention.output.LayerNorm.weight ----- False\nParameter: 142bert.encoder.layer.8.attention.output.LayerNorm.bias ----- False\nParameter: 143bert.encoder.layer.8.intermediate.dense.weight ----- False\nParameter: 144bert.encoder.layer.8.intermediate.dense.bias ----- False\nParameter: 145bert.encoder.layer.8.output.dense.weight ----- False\nParameter: 146bert.encoder.layer.8.output.dense.bias ----- False\nParameter: 147bert.encoder.layer.8.output.LayerNorm.weight ----- False\nParameter: 148bert.encoder.layer.8.output.LayerNorm.bias ----- False\nParameter: 149bert.encoder.layer.9.attention.self.query.weight ----- False\nParameter: 150bert.encoder.layer.9.attention.self.query.bias ----- False\nParameter: 151bert.encoder.layer.9.attention.self.key.weight ----- False\nParameter: 152bert.encoder.layer.9.attention.self.key.bias ----- False\nParameter: 153bert.encoder.layer.9.attention.self.value.weight ----- False\nParameter: 154bert.encoder.layer.9.attention.self.value.bias ----- False\nParameter: 155bert.encoder.layer.9.attention.output.dense.weight ----- False\nParameter: 156bert.encoder.layer.9.attention.output.dense.bias ----- False\nParameter: 157bert.encoder.layer.9.attention.output.LayerNorm.weight ----- False\nParameter: 158bert.encoder.layer.9.attention.output.LayerNorm.bias ----- False\nParameter: 159bert.encoder.layer.9.intermediate.dense.weight ----- False\nParameter: 160bert.encoder.layer.9.intermediate.dense.bias ----- False\nParameter: 161bert.encoder.layer.9.output.dense.weight ----- False\nParameter: 162bert.encoder.layer.9.output.dense.bias ----- False\nParameter: 163bert.encoder.layer.9.output.LayerNorm.weight ----- False\nParameter: 164bert.encoder.layer.9.output.LayerNorm.bias ----- False\nParameter: 165bert.encoder.layer.10.attention.self.query.weight ----- False\nParameter: 166bert.encoder.layer.10.attention.self.query.bias ----- False\nParameter: 167bert.encoder.layer.10.attention.self.key.weight ----- False\nParameter: 168bert.encoder.layer.10.attention.self.key.bias ----- False\nParameter: 169bert.encoder.layer.10.attention.self.value.weight ----- False\nParameter: 170bert.encoder.layer.10.attention.self.value.bias ----- False\nParameter: 171bert.encoder.layer.10.attention.output.dense.weight ----- False\nParameter: 172bert.encoder.layer.10.attention.output.dense.bias ----- False\nParameter: 173bert.encoder.layer.10.attention.output.LayerNorm.weight ----- False\nParameter: 174bert.encoder.layer.10.attention.output.LayerNorm.bias ----- False\nParameter: 175bert.encoder.layer.10.intermediate.dense.weight ----- False\nParameter: 176bert.encoder.layer.10.intermediate.dense.bias ----- False\nParameter: 177bert.encoder.layer.10.output.dense.weight ----- False\nParameter: 178bert.encoder.layer.10.output.dense.bias ----- False\nParameter: 179bert.encoder.layer.10.output.LayerNorm.weight ----- False\nParameter: 180bert.encoder.layer.10.output.LayerNorm.bias ----- False\nParameter: 181bert.encoder.layer.11.attention.self.query.weight ----- False\nParameter: 182bert.encoder.layer.11.attention.self.query.bias ----- False\nParameter: 183bert.encoder.layer.11.attention.self.key.weight ----- False\nParameter: 184bert.encoder.layer.11.attention.self.key.bias ----- False\nParameter: 185bert.encoder.layer.11.attention.self.value.weight ----- False\nParameter: 186bert.encoder.layer.11.attention.self.value.bias ----- False\nParameter: 187bert.encoder.layer.11.attention.output.dense.weight ----- False\nParameter: 188bert.encoder.layer.11.attention.output.dense.bias ----- False\nParameter: 189bert.encoder.layer.11.attention.output.LayerNorm.weight ----- False\nParameter: 190bert.encoder.layer.11.attention.output.LayerNorm.bias ----- False\nParameter: 191bert.encoder.layer.11.intermediate.dense.weight ----- False\nParameter: 192bert.encoder.layer.11.intermediate.dense.bias ----- False\nParameter: 193bert.encoder.layer.11.output.dense.weight ----- False\nParameter: 194bert.encoder.layer.11.output.dense.bias ----- False\nParameter: 195bert.encoder.layer.11.output.LayerNorm.weight ----- False\nParameter: 196bert.encoder.layer.11.output.LayerNorm.bias ----- False\nParameter: 197bert.pooler.dense.weight ----- False\nParameter: 198bert.pooler.dense.bias ----- False\nParameter: 199classifier.weight ----- True\nParameter: 200classifier.bias ----- True\n\n\n\n# Load model\nmodel_id = \"bert-base-cased\"\nmodel = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\n# Encoder block 10 starts at index 165 and\n# we freeze everything before that block\nfor index, (name, param) in enumerate(model.named_parameters()):\n    if index < 165:\n        param.requires_grad = False\n\n# Trainer which executes the training process\ntrainer = Trainer(\n   model=model,\n   args=training_args,\n   train_dataset=tokenized_train,\n   eval_dataset=tokenized_test,\n   tokenizer=tokenizer,\n   data_collator=data_collator,\n   compute_metrics=compute_metrics,\n)\ntrainer.train()\ntrainer.evaluate()\n\nSome weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\n\n\n    \n      \n      \n      [534/534 00:21, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n500\n0.474600\n\n\n\n\n\n\n\n    \n      \n      \n      [67/67 00:01]\n    \n    \n\n\n{'eval_loss': 0.4092540740966797,\n 'eval_f1': 0.8141086749285034,\n 'eval_runtime': 2.7437,\n 'eval_samples_per_second': 388.523,\n 'eval_steps_per_second': 24.419,\n 'epoch': 1.0}\n\n\n\n\n[BONUS] Freeze blocks\n\n# scores = []\n# for index in range(12):\n#     # Re-load model\n#     model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=2)\n#     tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n\n#     # Freeze encoder blocks 0-index\n#     for name, param in model.named_parameters():\n#         if \"layer\" in name:\n#             layer_nr = int(name.split(\"layer\")[1].split(\".\")[1])\n#             if layer_nr <= index:\n#                 param.requires_grad = False\n#         else:\n#             param.requires_grad = True\n\n#     # Train\n#     trainer = Trainer(\n#       model=model,\n#       args=training_args,\n#       train_dataset=tokenized_train,\n#       eval_dataset=tokenized_test,\n#       tokenizer=tokenizer,\n#       data_collator=data_collator,\n#       compute_metrics=compute_metrics,\n#     )\n#     trainer.train()\n\n#     # Evaluate\n#     score = trainer.evaluate()[\"eval_f1\"]\n#     scores.append(score)\n\n\n# scores\n\n[0.8541862652869239,\n 0.8525519848771267,\n 0.8514664143803217,\n 0.8506616257088847,\n 0.8398104265402844,\n 0.8391345249294448,\n 0.8377358490566037,\n 0.8433962264150944,\n 0.8258801141769743,\n 0.816247582205029,\n 0.7917485265225934,\n 0.7019400352733686]\n\n\n\n# import matplotlib.pyplot as plt\n# import numpy as np\n\n# # Create Figure\n# plt.figure(figsize=(8,4))\n\n# # Prepare Data\n# x = [f\"0-{index}\" for index in range(12)]\n# x[0] = \"None\"\n# x[-1] = \"All\"\n# y = [\n#     0.8541862652869239,\n#     0.8525519848771267,\n#     0.8514664143803217,\n#     0.8506616257088847,\n#     0.8398104265402844,\n#     0.8391345249294448,\n#     0.8377358490566037,\n#     0.8433962264150944,\n#     0.8258801141769743,\n#     0.816247582205029,\n#     0.7917485265225934,\n#     0.7019400352733686\n# ][::-1]\n\n# # Stylize Figure\n# plt.grid(color='#ECEFF1')\n# plt.axvline(x=4, color=\"#EC407A\", linestyle=\"--\")\n# plt.title(\"Effect of Frozen Encoder Blocks on Training Performance\")\n# plt.ylabel(\"F1-score\")\n# plt.xlabel(\"Trainable encoder blocks\")\n\n# # Plot Data\n# plt.plot(x, y, color=\"black\")\n\n# # Additional Annotation\n# plt.annotate(\n#     'Performance stabilizing',\n#     xy=(4, y[4]),\n#     xytext=(4.5, y[4]-.05),\n#     arrowprops=dict(\n#         arrowstyle=\"-|>\",\n#         connectionstyle=\"arc3\",\n#         color=\"#00ACC1\")\n# )\n# plt.savefig(\"multiple_frozen_blocks.png\", dpi=300, bbox_inches='tight')",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#few-shot-classification",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#few-shot-classification",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "Few-shot Classification",
    "text": "Few-shot Classification\n\nfrom setfit import sample_dataset\n\n# We simulate a few-shot setting by sampling 16 examples per class\nsampled_train_data = sample_dataset(tomatoes[\"train\"], num_samples=16)\n\n\nfrom setfit import SetFitModel\n\n# Load a pre-trained SentenceTransformer model\nmodel = SetFitModel.from_pretrained(\"sentence-transformers/all-mpnet-base-v2\")\n\n/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n  and should_run_async(code)\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\nmodel_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.\n\n\n\nfrom setfit import TrainingArguments as SetFitTrainingArguments\nfrom setfit import Trainer as SetFitTrainer\n\n# Define training arguments\nargs = SetFitTrainingArguments(\n    num_epochs=3, # The number of epochs to use for contrastive learning\n    num_iterations=20  # The number of text pairs to generate\n)\nargs.eval_strategy = args.evaluation_strategy\n\n# Create trainer\ntrainer = SetFitTrainer(\n    model=model,\n    args=args,\n    train_dataset=sampled_train_data,\n    eval_dataset=test_data,\n    metric=\"f1\"\n)\n\n\n\n\n\n# from setfit import SetFitTrainer\n\n# # Create trainer\n# trainer = SetFitTrainer(\n#     model=model,\n#     train_dataset=sampled_train_data,\n#     eval_dataset=test_data,\n#     metric=\"f1\",\n#     num_epochs=3, # The number of epochs to use for contrastive learning\n# )\n\n\n# Training loop\ntrainer.train()\n\n***** Running training *****\n  Num unique pairs = 1280\n  Batch size = 16\n  Num epochs = 3\n  Total optimization steps = 240\n\n\n\n    \n      \n      \n      [240/240 00:37, Epoch 3/0]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n\n\n\n\n\n# Evaluate the model on our test data\ntrainer.evaluate()\n\n***** Running evaluation *****\n\n\n{'f1': 0.8363988383349468}\n\n\n\nmodel.model_head\n\n/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n  and should_run_async(code)\n\n\nLogisticRegression()In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.LogisticRegressionLogisticRegression()",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#mlm",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#mlm",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "MLM",
    "text": "MLM\n\nfrom transformers import AutoTokenizer, AutoModelForMaskedLM\n\n# Load model for Masked Language Modeling (MLM)\nmodel = AutoModelForMaskedLM.from_pretrained(\"bert-base-cased\")\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n\nSome weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n\ndef preprocess_function(examples):\n   return tokenizer(examples[\"text\"], truncation=True)\n\n# Tokenize data\ntokenized_train = train_data.map(preprocess_function, batched=True)\ntokenized_train = tokenized_train.remove_columns(\"label\")\ntokenized_test = test_data.map(preprocess_function, batched=True)\ntokenized_test = tokenized_test.remove_columns(\"label\")\n\n\n\n\n\n\n\n\nfrom transformers import DataCollatorForLanguageModeling\n\n# Masking Tokens\ndata_collator = DataCollatorForLanguageModeling(\n    tokenizer=tokenizer,\n    mlm=True,\n    mlm_probability=0.15\n)\n\n\n# from transformers import DataCollatorForWholeWordMask\n\n# # Masking Whole Words\n# data_collator = DataCollatorForWholeWordMask(\n#     tokenizer=tokenizer,\n#     mlm=True,\n#     mlm_probability=0.15\n# )\n\n\n# Training arguments for parameter tuning\ntraining_args = TrainingArguments(\n   \"model\",\n   learning_rate=2e-5,\n   per_device_train_batch_size=16,\n   per_device_eval_batch_size=16,\n   num_train_epochs=10,\n   weight_decay=0.01,\n   save_strategy=\"epoch\",\n   report_to=\"none\"\n)\n\n\n# Initialize Trainer\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_train,\n    eval_dataset=tokenized_test,\n    tokenizer=tokenizer,\n    data_collator=data_collator\n)\n\n\n# Save pre-trained tokenizer\ntokenizer.save_pretrained(\"mlm\")\n\n# Train model\ntrainer.train()\n\n# Save updated model\nmodel.save_pretrained(\"mlm\")\n\n\n    \n      \n      \n      [5340/5340 12:10, Epoch 10/10]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n500\n2.601700\n\n\n1000\n2.377500\n\n\n1500\n2.313100\n\n\n2000\n2.187500\n\n\n2500\n2.150400\n\n\n3000\n2.096100\n\n\n3500\n2.059500\n\n\n4000\n1.990300\n\n\n4500\n1.986100\n\n\n5000\n1.958500\n\n\n\n\n\n\n\nfrom transformers import pipeline\n\n# Load and create predictions\nmask_filler = pipeline(\"fill-mask\", model=\"bert-base-cased\")\npreds = mask_filler(\"What a horrible [MASK]!\")\n\n# Print results\nfor pred in preds:\n    print(f\">>> {pred['sequence']}\")\n\n/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n  and should_run_async(code)\nSome weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n>>> What a horrible idea!\n>>> What a horrible dream!\n>>> What a horrible thing!\n>>> What a horrible day!\n>>> What a horrible thought!\n\n\n\n# Load and create predictions\nmask_filler = pipeline(\"fill-mask\", model=\"mlm\")\npreds = mask_filler(\"What a horrible [MASK]!\")\n\n# Print results\nfor pred in preds:\n    print(f\">>> {pred['sequence']}\")\n\n>>> What a horrible movie!\n>>> What a horrible film!\n>>> What a horrible mess!\n>>> What a horrible comedy!\n>>> What a horrible story!",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#named-entity-recognition",
    "href": "Hands-On-Large-Language-Models/chapter11/Chapter 11 - Fine-Tuning BERT.html#named-entity-recognition",
    "title": "Chapter 11 - Fine-Tuning BERT",
    "section": "Named Entity Recognition",
    "text": "Named Entity Recognition\nHere are a number of interesting datasets you can also explore for NER: * tner/mit_movie_trivia * tner/mit_restaurant * wnut_17 * conll2003\n\nfrom transformers import AutoModelForTokenClassification, AutoTokenizer\nfrom transformers import DataCollatorWithPadding\nfrom transformers import TrainingArguments, Trainer\nimport numpy as np\n\n\n# The CoNLL-2003 dataset for NER\ndataset = load_dataset(\"conll2003\", trust_remote_code=True)\n\n\n\n\n\n\n\nThe repository for conll2003 contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/conll2003.\nYou can avoid this prompt in future by passing the argument `trust_remote_code=True`.\n\nDo you wish to run the custom code? [y/N] y\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nexample = dataset[\"train\"][848]\nexample\n\n{'id': '848',\n 'tokens': ['Dean',\n  'Palmer',\n  'hit',\n  'his',\n  '30th',\n  'homer',\n  'for',\n  'the',\n  'Rangers',\n  '.'],\n 'pos_tags': [22, 22, 38, 29, 16, 21, 15, 12, 23, 7],\n 'chunk_tags': [11, 12, 21, 11, 12, 12, 13, 11, 12, 0],\n 'ner_tags': [1, 2, 0, 0, 0, 0, 0, 0, 3, 0]}\n\n\n\nlabel2id = {\n    'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4,\n    'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8\n}\nid2label = {index: label for label, index in label2id.items()}\nlabel2id\n\n{'O': 0,\n 'B-PER': 1,\n 'I-PER': 2,\n 'B-ORG': 3,\n 'I-ORG': 4,\n 'B-LOC': 5,\n 'I-LOC': 6,\n 'B-MISC': 7,\n 'I-MISC': 8}\n\n\n\nfrom transformers import AutoModelForTokenClassification\n\n# Load tokenizer\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n\n# Load model\nmodel = AutoModelForTokenClassification.from_pretrained(\n    \"bert-base-cased\",\n    num_labels=len(id2label),\n    id2label=id2label,\n    label2id=label2id\n)\n\nSome weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\n\n\n# Split individual tokens into sub-tokens\ntoken_ids = tokenizer(example[\"tokens\"], is_split_into_words=True)[\"input_ids\"]\nsub_tokens = tokenizer.convert_ids_to_tokens(token_ids)\nsub_tokens\n\n['[CLS]',\n 'Dean',\n 'Palmer',\n 'hit',\n 'his',\n '30th',\n 'home',\n '##r',\n 'for',\n 'the',\n 'Rangers',\n '.',\n '[SEP]']\n\n\n\ndef align_labels(examples):\n    token_ids = tokenizer(examples[\"tokens\"], truncation=True, is_split_into_words=True)\n    labels = examples[\"ner_tags\"]\n\n    updated_labels = []\n    for index, label in enumerate(labels):\n\n        # Map tokens to their respective word\n        word_ids = token_ids.word_ids(batch_index=index)\n        previous_word_idx = None\n        label_ids = []\n        for word_idx in word_ids:\n\n            # The start of a new word\n            if word_idx != previous_word_idx:\n\n                previous_word_idx = word_idx\n                updated_label = -100 if word_idx is None else label[word_idx]\n                label_ids.append(updated_label)\n\n            # Special token is -100\n            elif word_idx is None:\n                label_ids.append(-100)\n\n            # If the label is B-XXX we change it to I-XXX\n            else:\n                updated_label = label[word_idx]\n                if updated_label % 2 == 1:\n                    updated_label += 1\n                label_ids.append(updated_label)\n\n        updated_labels.append(label_ids)\n\n    token_ids[\"labels\"] = updated_labels\n    return token_ids\n\ntokenized = dataset.map(align_labels, batched=True)\n\n\n\n\n\n\n\n\n\n\n\n# Difference between original and updated labels\nprint(f\"Original: {example['ner_tags']}\")\nprint(f\"Updated: {tokenized['train'][848]['labels']}\")\n\nOriginal: [1, 2, 0, 0, 0, 0, 0, 0, 3, 0]\nUpdated: [-100, 1, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, -100]\n\n\n\nimport evaluate\n\n# Load sequential evaluation\nseqeval = evaluate.load(\"seqeval\")\n\ndef compute_metrics(eval_pred):\n    # Create predictions\n    logits, labels = eval_pred\n    predictions = np.argmax(logits, axis=2)\n\n    true_predictions = []\n    true_labels = []\n\n    # Document-level iteration\n    for prediction, label in zip(predictions, labels):\n\n      # token-level iteration\n      for token_prediction, token_label in zip(prediction, label):\n\n        # We ignore special tokens\n        if token_label != -100:\n          true_predictions.append([id2label[token_prediction]])\n          true_labels.append([id2label[token_label]])\n\n    results = seqeval.compute(predictions=true_predictions, references=true_labels)\n    return {\"f1\": results[\"overall_f1\"]}\n\n\nfrom transformers import DataCollatorForTokenClassification\n\n# Token-classification Data Collator\ndata_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)\n\n\n# Training arguments for parameter tuning\ntraining_args = TrainingArguments(\n   \"model\",\n   learning_rate=2e-5,\n   per_device_train_batch_size=16,\n   per_device_eval_batch_size=16,\n   num_train_epochs=1,\n   weight_decay=0.01,\n   save_strategy=\"epoch\",\n   report_to=\"none\"\n)\n\n# Initialize Trainer\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized[\"train\"],\n    eval_dataset=tokenized[\"test\"],\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n)\ntrainer.train()\n\n\n    \n      \n      \n      [878/878 02:49, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n500\n0.047500\n\n\n\n\n\n\nTrainOutput(global_step=878, training_loss=0.04094860494001037, metrics={'train_runtime': 169.4752, 'train_samples_per_second': 82.85, 'train_steps_per_second': 5.181, 'total_flos': 351240792638148.0, 'train_loss': 0.04094860494001037, 'epoch': 1.0})\n\n\n\n# Evaluate the model on our test data\ntrainer.evaluate()\n\n\n    \n      \n      \n      [216/216 00:09]\n    \n    \n\n\n{'eval_loss': 0.16888542473316193,\n 'eval_f1': 0.9180087380808113,\n 'eval_runtime': 14.5731,\n 'eval_samples_per_second': 236.943,\n 'eval_steps_per_second': 14.822,\n 'epoch': 1.0}\n\n\n\nfrom transformers import pipeline\n\n# Save our fine-tuned model\ntrainer.save_model(\"ner_model\")\n\n# Run inference on the fine-tuned model\ntoken_classifier = pipeline(\n    \"token-classification\",\n    model=\"ner_model\",\n)\ntoken_classifier(\"My name is Maarten.\")\n\n[{'entity': 'B-PER',\n  'score': 0.99534035,\n  'index': 4,\n  'word': 'Ma',\n  'start': 11,\n  'end': 13},\n {'entity': 'I-PER',\n  'score': 0.9928328,\n  'index': 5,\n  'word': '##arte',\n  'start': 13,\n  'end': 17},\n {'entity': 'I-PER',\n  'score': 0.9954301,\n  'index': 6,\n  'word': '##n',\n  'start': 17,\n  'end': 18}]",
    "crumbs": [
      "Home",
      "Training",
      "11. Fine-tuning BERT"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html",
    "href": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html",
    "title": "Chapter 9 - Multimodal Large Language Models",
    "section": "",
    "text": "from urllib.request import urlopen\nfrom PIL import Image\n\n# Load an AI-generated image of a puppy playing in the snow\npuppy_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/puppy.png\"\nimage = Image.open(urlopen(puppy_path)).convert(\"RGB\")\ncaption = \"a puppy playing in the snow\"\n\n\nimage\n\n\n\n\n\n\n\n\n\n\n\nfrom transformers import CLIPTokenizerFast, CLIPProcessor, CLIPModel\n\nmodel_id = \"openai/clip-vit-base-patch32\"\n\n# Load a tokenizer to preprocess the text\nclip_tokenizer = CLIPTokenizerFast.from_pretrained(model_id)\n\n# Load a processor to preprocess the images\nclip_processor = CLIPProcessor.from_pretrained(model_id)\n\n# Main model for generating text and image embeddings\nmodel = CLIPModel.from_pretrained(model_id)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\n\n\n\n\n\n\n\n# Tokenize our input\ninputs = clip_tokenizer(caption, return_tensors=\"pt\")\ninputs\n\n{'input_ids': tensor([[49406,   320,  6829,  1629,   530,   518,  2583, 49407]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}\n\n\n\n# Convert our input back to tokens\nclip_tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0])\n\n['<|startoftext|>',\n 'a</w>',\n 'puppy</w>',\n 'playing</w>',\n 'in</w>',\n 'the</w>',\n 'snow</w>',\n '<|endoftext|>']\n\n\n\n# Create a text embedding\ntext_embedding = model.get_text_features(**inputs)\ntext_embedding.shape\n\ntorch.Size([1, 512])\n\n\n\n# Preprocess image\nprocessed_image = clip_processor(\n    text=None, images=image, return_tensors='pt'\n)['pixel_values']\n\nprocessed_image.shape\n\ntorch.Size([1, 3, 224, 224])\n\n\n\nimport torch\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Prepare image for visualization\nimg = processed_image.squeeze(0)\nimg = img.permute(*torch.arange(img.ndim - 1, -1, -1))\nimg = np.einsum('ijk->jik', img)\n\n# Visualize preprocessed image\nplt.imshow(img)\nplt.axis('off')\n\nWARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n\n\n\n\n\n\n\n\n\n\n# Create the image embedding\nimage_embedding = model.get_image_features(processed_image)\nimage_embedding.shape\n\ntorch.Size([1, 512])\n\n\n\n# Normalize the embeddings\ntext_embedding /= text_embedding.norm(dim=-1, keepdim=True)\nimage_embedding /= image_embedding.norm(dim=-1, keepdim=True)\n\n# Calculate their similarity\ntext_embedding = text_embedding.detach().cpu().numpy()\nimage_embedding = image_embedding.detach().cpu().numpy()\nscore = text_embedding @ image_embedding.T\nscore\n\narray([[0.33149636]], dtype=float32)\n\n\n\n\n\n\nfrom urllib.request import urlopen\nfrom PIL import Image\n\n# Load an AI-generated image of a puppy playing in the snow\ncat_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/cat.png\"\ncar_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png\"\npaths = [puppy_path, cat_path, car_path]\nimages = [Image.open(urlopen(path)).convert(\"RGBA\") for path in paths]\ncaptions = [\n    \"a puppy playing in the snow\",\n    \"a pixelated image of a cute cat\",\n    \"A supercar on the road \\nwith the sunset in the background\"\n]\n\nimport numpy as np\n\n# Embed all images\nimage_embeddings = []\nfor image in images:\n  image_processed = clip_processor(images=image, return_tensors='pt')['pixel_values']\n  image_embedding = model.get_image_features(image_processed).detach().cpu().numpy()[0]\n  image_embeddings.append(image_embedding)\nimage_embeddings = np.array(image_embeddings)\n\n# Embed all captions\ntext_embeddings = []\nfor caption in captions:\n  inputs = clip_tokenizer(caption, return_tensors=\"pt\")\n  text_emb = model.get_text_features(**inputs).detach().cpu().numpy()[0]\n  text_embeddings.append(text_emb)\ntext_embeddings = np.array(text_embeddings)\n\n\n# Calculate cosine similarity between images and captions\nfrom sklearn.metrics.pairwise import cosine_similarity\nsim_matrix = cosine_similarity(image_embeddings, text_embeddings)\n\n\n# Create base figure\nplt.figure(figsize=(20, 14))\nplt.imshow(sim_matrix, cmap='viridis')\n\n# Adjust ticks with correct labels\nplt.yticks(range(len(captions)), captions, fontsize=18)\nplt.xticks([])\n\n# Visualize\nfor i, image in enumerate(images):\n    plt.imshow(image, extent=(i - 0.5, i + 0.5, -1.6, -0.6), origin=\"lower\")\n\n# Add the captions at the correct indices\nfor x in range(sim_matrix.shape[1]):\n    for y in range(sim_matrix.shape[0]):\n        plt.text(x, y, f\"{sim_matrix[y, x]:.2f}\", ha=\"center\", va=\"center\", size=30)\n\n# Remove unnecessary spines\nfor side in [\"left\", \"top\", \"right\", \"bottom\"]:\n  plt.gca().spines[side].set_visible(False)\n\n# Resize blocks\nplt.xlim([-0.5, len(captions) - 0.5])\nplt.ylim([len(captions) + 0.5, -2])\n# plt.title(\"Similarity Matrix\", size=20)\nplt.savefig(\"sim_matrix.png\", dpi=300, bbox_inches='tight')\n\n\n\n\n\n\n\n\n\n\n\n\nfrom sentence_transformers import SentenceTransformer, util\n\n# Load SBERT-compatible CLIP model\nmodel = SentenceTransformer('clip-ViT-B-32')\n\n# Encode the images\nimage_embeddings = model.encode(images)\n\n# Encode the captions\ntext_embeddings = model.encode(captions)\n\n#Compute cosine similarities\nsim_matrix = util.cos_sim(image_embeddings, text_embeddings)\nprint(sim_matrix)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ntensor([[0.3315, 0.1863, 0.1084],\n        [0.1488, 0.3463, 0.0947],\n        [0.0762, 0.1260, 0.3098]])",
    "crumbs": [
      "Home",
      "Applications",
      "9. Multimodal LLMs"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html#clip",
    "href": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html#clip",
    "title": "Chapter 9 - Multimodal Large Language Models",
    "section": "",
    "text": "from urllib.request import urlopen\nfrom PIL import Image\n\n# Load an AI-generated image of a puppy playing in the snow\npuppy_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/puppy.png\"\nimage = Image.open(urlopen(puppy_path)).convert(\"RGB\")\ncaption = \"a puppy playing in the snow\"\n\n\nimage\n\n\n\n\n\n\n\n\n\n\n\nfrom transformers import CLIPTokenizerFast, CLIPProcessor, CLIPModel\n\nmodel_id = \"openai/clip-vit-base-patch32\"\n\n# Load a tokenizer to preprocess the text\nclip_tokenizer = CLIPTokenizerFast.from_pretrained(model_id)\n\n# Load a processor to preprocess the images\nclip_processor = CLIPProcessor.from_pretrained(model_id)\n\n# Main model for generating text and image embeddings\nmodel = CLIPModel.from_pretrained(model_id)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\n\n\n\n\n\n\n\n# Tokenize our input\ninputs = clip_tokenizer(caption, return_tensors=\"pt\")\ninputs\n\n{'input_ids': tensor([[49406,   320,  6829,  1629,   530,   518,  2583, 49407]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}\n\n\n\n# Convert our input back to tokens\nclip_tokenizer.convert_ids_to_tokens(inputs[\"input_ids\"][0])\n\n['<|startoftext|>',\n 'a</w>',\n 'puppy</w>',\n 'playing</w>',\n 'in</w>',\n 'the</w>',\n 'snow</w>',\n '<|endoftext|>']\n\n\n\n# Create a text embedding\ntext_embedding = model.get_text_features(**inputs)\ntext_embedding.shape\n\ntorch.Size([1, 512])\n\n\n\n# Preprocess image\nprocessed_image = clip_processor(\n    text=None, images=image, return_tensors='pt'\n)['pixel_values']\n\nprocessed_image.shape\n\ntorch.Size([1, 3, 224, 224])\n\n\n\nimport torch\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Prepare image for visualization\nimg = processed_image.squeeze(0)\nimg = img.permute(*torch.arange(img.ndim - 1, -1, -1))\nimg = np.einsum('ijk->jik', img)\n\n# Visualize preprocessed image\nplt.imshow(img)\nplt.axis('off')\n\nWARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n\n\n\n\n\n\n\n\n\n\n# Create the image embedding\nimage_embedding = model.get_image_features(processed_image)\nimage_embedding.shape\n\ntorch.Size([1, 512])\n\n\n\n# Normalize the embeddings\ntext_embedding /= text_embedding.norm(dim=-1, keepdim=True)\nimage_embedding /= image_embedding.norm(dim=-1, keepdim=True)\n\n# Calculate their similarity\ntext_embedding = text_embedding.detach().cpu().numpy()\nimage_embedding = image_embedding.detach().cpu().numpy()\nscore = text_embedding @ image_embedding.T\nscore\n\narray([[0.33149636]], dtype=float32)\n\n\n\n\n\n\nfrom urllib.request import urlopen\nfrom PIL import Image\n\n# Load an AI-generated image of a puppy playing in the snow\ncat_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/cat.png\"\ncar_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png\"\npaths = [puppy_path, cat_path, car_path]\nimages = [Image.open(urlopen(path)).convert(\"RGBA\") for path in paths]\ncaptions = [\n    \"a puppy playing in the snow\",\n    \"a pixelated image of a cute cat\",\n    \"A supercar on the road \\nwith the sunset in the background\"\n]\n\nimport numpy as np\n\n# Embed all images\nimage_embeddings = []\nfor image in images:\n  image_processed = clip_processor(images=image, return_tensors='pt')['pixel_values']\n  image_embedding = model.get_image_features(image_processed).detach().cpu().numpy()[0]\n  image_embeddings.append(image_embedding)\nimage_embeddings = np.array(image_embeddings)\n\n# Embed all captions\ntext_embeddings = []\nfor caption in captions:\n  inputs = clip_tokenizer(caption, return_tensors=\"pt\")\n  text_emb = model.get_text_features(**inputs).detach().cpu().numpy()[0]\n  text_embeddings.append(text_emb)\ntext_embeddings = np.array(text_embeddings)\n\n\n# Calculate cosine similarity between images and captions\nfrom sklearn.metrics.pairwise import cosine_similarity\nsim_matrix = cosine_similarity(image_embeddings, text_embeddings)\n\n\n# Create base figure\nplt.figure(figsize=(20, 14))\nplt.imshow(sim_matrix, cmap='viridis')\n\n# Adjust ticks with correct labels\nplt.yticks(range(len(captions)), captions, fontsize=18)\nplt.xticks([])\n\n# Visualize\nfor i, image in enumerate(images):\n    plt.imshow(image, extent=(i - 0.5, i + 0.5, -1.6, -0.6), origin=\"lower\")\n\n# Add the captions at the correct indices\nfor x in range(sim_matrix.shape[1]):\n    for y in range(sim_matrix.shape[0]):\n        plt.text(x, y, f\"{sim_matrix[y, x]:.2f}\", ha=\"center\", va=\"center\", size=30)\n\n# Remove unnecessary spines\nfor side in [\"left\", \"top\", \"right\", \"bottom\"]:\n  plt.gca().spines[side].set_visible(False)\n\n# Resize blocks\nplt.xlim([-0.5, len(captions) - 0.5])\nplt.ylim([len(captions) + 0.5, -2])\n# plt.title(\"Similarity Matrix\", size=20)\nplt.savefig(\"sim_matrix.png\", dpi=300, bbox_inches='tight')\n\n\n\n\n\n\n\n\n\n\n\n\nfrom sentence_transformers import SentenceTransformer, util\n\n# Load SBERT-compatible CLIP model\nmodel = SentenceTransformer('clip-ViT-B-32')\n\n# Encode the images\nimage_embeddings = model.encode(images)\n\n# Encode the captions\ntext_embeddings = model.encode(captions)\n\n#Compute cosine similarities\nsim_matrix = util.cos_sim(image_embeddings, text_embeddings)\nprint(sim_matrix)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ntensor([[0.3315, 0.1863, 0.1084],\n        [0.1488, 0.3463, 0.0947],\n        [0.0762, 0.1260, 0.3098]])",
    "crumbs": [
      "Home",
      "Applications",
      "9. Multimodal LLMs"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html#blip-2",
    "href": "Hands-On-Large-Language-Models/chapter09/Chapter 9 - Multimodal Large Language Models.html#blip-2",
    "title": "Chapter 9 - Multimodal Large Language Models",
    "section": "BLIP-2",
    "text": "BLIP-2\n\nfrom transformers import AutoProcessor, Blip2ForConditionalGeneration\nimport torch\n\n# Load processor and main model\nblip_processor = AutoProcessor.from_pretrained(\n    \"Salesforce/blip2-opt-2.7b\",\n    revision=\"51572668da0eb669e01a189dc22abe6088589a24\"  # Choose specific model because of: https://huggingface.co/Salesforce/blip2-opt-2.7b/discussions/39\n)\nmodel = Blip2ForConditionalGeneration.from_pretrained(\n    \"Salesforce/blip2-opt-2.7b\",\n    revision=\"51572668da0eb669e01a189dc22abe6088589a24\",\n    torch_dtype=torch.float16\n)\n\n# Send the model to GPU to speed up inference\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel.to(device)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBlip2ForConditionalGeneration(\n  (vision_model): Blip2VisionModel(\n    (embeddings): Blip2VisionEmbeddings(\n      (patch_embedding): Conv2d(3, 1408, kernel_size=(14, 14), stride=(14, 14))\n    )\n    (encoder): Blip2Encoder(\n      (layers): ModuleList(\n        (0-38): 39 x Blip2EncoderLayer(\n          (self_attn): Blip2Attention(\n            (dropout): Dropout(p=0.0, inplace=False)\n            (qkv): Linear(in_features=1408, out_features=4224, bias=True)\n            (projection): Linear(in_features=1408, out_features=1408, bias=True)\n          )\n          (layer_norm1): LayerNorm((1408,), eps=1e-06, elementwise_affine=True)\n          (mlp): Blip2MLP(\n            (activation_fn): GELUActivation()\n            (fc1): Linear(in_features=1408, out_features=6144, bias=True)\n            (fc2): Linear(in_features=6144, out_features=1408, bias=True)\n          )\n          (layer_norm2): LayerNorm((1408,), eps=1e-06, elementwise_affine=True)\n        )\n      )\n    )\n    (post_layernorm): LayerNorm((1408,), eps=1e-06, elementwise_affine=True)\n  )\n  (qformer): Blip2QFormerModel(\n    (layernorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n    (encoder): Blip2QFormerEncoder(\n      (layer): ModuleList(\n        (0): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (1): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (2): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (3): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (4): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (5): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (6): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (7): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (8): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (9): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (10): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (crossattention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=1408, out_features=768, bias=True)\n              (value): Linear(in_features=1408, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (11): Blip2QFormerLayer(\n          (attention): Blip2QFormerAttention(\n            (attention): Blip2QFormerMultiHeadAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): Blip2QFormerSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate_query): Blip2QFormerIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output_query): Blip2QFormerOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (language_projection): Linear(in_features=768, out_features=2560, bias=True)\n  (language_model): OPTForCausalLM(\n    (model): OPTModel(\n      (decoder): OPTDecoder(\n        (embed_tokens): Embedding(50272, 2560, padding_idx=1)\n        (embed_positions): OPTLearnedPositionalEmbedding(2050, 2560)\n        (final_layer_norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)\n        (layers): ModuleList(\n          (0-31): 32 x OPTDecoderLayer(\n            (self_attn): OPTAttention(\n              (k_proj): Linear(in_features=2560, out_features=2560, bias=True)\n              (v_proj): Linear(in_features=2560, out_features=2560, bias=True)\n              (q_proj): Linear(in_features=2560, out_features=2560, bias=True)\n              (out_proj): Linear(in_features=2560, out_features=2560, bias=True)\n            )\n            (activation_fn): ReLU()\n            (self_attn_layer_norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)\n            (fc1): Linear(in_features=2560, out_features=10240, bias=True)\n            (fc2): Linear(in_features=10240, out_features=2560, bias=True)\n            (final_layer_norm): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)\n          )\n        )\n      )\n    )\n    (lm_head): Linear(in_features=2560, out_features=50272, bias=False)\n  )\n)\n\n\n\nPreprocessing Images\n\n# Load image of a supercar\ncar_path = \"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png\"\nimage = Image.open(urlopen(car_path)).convert(\"RGB\")\nimage\n\n\n\n\n\n\n\n\n\n# Preprocess the image\ninputs = blip_processor(image, return_tensors=\"pt\").to(device, torch.float16)\ninputs[\"pixel_values\"].shape\n\ntorch.Size([1, 3, 224, 224])\n\n\n\nfrom sklearn.preprocessing import MinMaxScaler\n\n# Convert to numpy and go from (1, 3, 224, 224) to (224, 224, 3) in shape\nimage_inputs = inputs[\"pixel_values\"][0].detach().cpu().numpy()\nimage_inputs = np.einsum('ijk->kji', image_inputs)\nimage_inputs = np.einsum('ijk->jik', image_inputs)\n\n# Scale image inputs to 0-255 to represent RGB values\nscaler = MinMaxScaler(feature_range=(0, 255))\nimage_inputs = scaler.fit_transform(image_inputs.reshape(-1, image_inputs.shape[-1])).reshape(image_inputs.shape)\nimage_inputs = np.array(image_inputs, dtype=np.uint8)\n\n# Convert numpy array to Image\nImage.fromarray(image_inputs)\n\n\n\n\n\n\n\n\n\n\nPreprocessing Text\n\nblip_processor.tokenizer\n\nGPT2TokenizerFast(name_or_path='Salesforce/blip2-opt-2.7b', vocab_size=50265, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '</s>', 'eos_token': '</s>', 'unk_token': '</s>', 'pad_token': '<pad>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={\n    1: AddedToken(\"<pad>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n    2: AddedToken(\"</s>\", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n}\n\n\n\n# Preprocess the text\ntext = \"Her vocalization was remarkably melodic\"\ntoken_ids = blip_processor(image, text=text, return_tensors=\"pt\")\ntoken_ids = token_ids.to(device, torch.float16)[\"input_ids\"][0]\n\n# Convert input ids back to tokens\ntokens = blip_processor.tokenizer.convert_ids_to_tokens(token_ids)\ntokens\n\n['</s>', 'Her', 'Ġvocal', 'ization', 'Ġwas', 'Ġremarkably', 'Ġmel', 'odic']\n\n\n\n# Replace the space token with an underscore\ntokens = [token.replace(\"Ġ\", \"_\") for token in tokens]\ntokens\n\n['</s>', 'Her', '_vocal', 'ization', '_was', '_remarkably', '_mel', 'odic']\n\n\n\n\nUse Case 1: Image Captioning\n\n# Load an AI-generated image of a supercar\nimage = Image.open(urlopen(car_path)).convert(\"RGB\")\n\n# Convert an image into inputs and preprocess it\ninputs = blip_processor(image, return_tensors=\"pt\").to(device, torch.float16)\nimage\n\n\n\n\n\n\n\n\n\n# Generate image ids to be passed to the decoder (LLM)\ngenerated_ids = model.generate(**inputs, max_new_tokens=20)\n\n# Generate text from the image ids\ngenerated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)\ngenerated_text = generated_text[0].strip()\ngenerated_text\n\n/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py:456: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)\n  return F.conv2d(input, weight, bias, self.stride,\nBoth `max_new_tokens` (=20) and `max_length`(=51) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n\n\n'an orange supercar driving on the road at sunset'\n\n\n\nurl = \"https://upload.wikimedia.org/wikipedia/commons/7/70/Rorschach_blot_01.jpg\"\nimage = Image.open(urlopen(url)).convert(\"RGB\")\nimage\n\n\n\n\n\n\n\n\n\n# Load rorschach image\nurl = \"https://upload.wikimedia.org/wikipedia/commons/7/70/Rorschach_blot_01.jpg\"\nimage = Image.open(urlopen(url)).convert(\"RGB\")\n\n# Generate caption\ninputs = blip_processor(image, return_tensors=\"pt\").to(device, torch.float16)\ngenerated_ids = model.generate(**inputs, max_new_tokens=20)\ngenerated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)\ngenerated_text = generated_text[0].strip()\ngenerated_text\n\nBoth `max_new_tokens` (=20) and `max_length`(=51) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n\n\n'a black and white ink drawing of a bat'\n\n\n\n\nUse Case 2: Visual Question Answering\n\n# Load an AI-generated image of a supercar\nimage = Image.open(urlopen(car_path)).convert(\"RGB\")\n\n\n# Visual Question Answering\nprompt = \"Question: Write down what you see in this picture. Answer:\"\n\n# Process both the image and the prompt\ninputs = blip_processor(image, text=prompt, return_tensors=\"pt\").to(device, torch.float16)\n\n# Generate text\ngenerated_ids = model.generate(**inputs, max_new_tokens=30)\ngenerated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)\ngenerated_text = generated_text[0].strip()\ngenerated_text\n\nBoth `max_new_tokens` (=30) and `max_length`(=51) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n\n\n'A sports car driving on the road at sunset'\n\n\n\n# Chat-like prompting\nprompt = \"Question: Write down what you see in this picture. Answer: A sports car driving on the road at sunset. Question: What would it cost me to drive that car? Answer:\"\n\n# Generate output\ninputs = blip_processor(image, text=prompt, return_tensors=\"pt\").to(device, torch.float16)\ngenerated_ids = model.generate(**inputs, max_new_tokens=30)\ngenerated_text = blip_processor.batch_decode(generated_ids, skip_special_tokens=True)\ngenerated_text = generated_text[0].strip()\ngenerated_text\n\nBoth `max_new_tokens` (=30) and `max_length`(=51) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n\n\n'$1,000,000'\n\n\n\nfrom IPython.display import HTML, display\nimport ipywidgets as widgets\n\ndef text_eventhandler(*args):\n  question = args[0][\"new\"]\n  if question:\n    args[0][\"owner\"].value = \"\"\n\n    # Create prompt\n    if not memory:\n      prompt = \" Question: \" + question + \" Answer:\"\n    else:\n      template = \"Question: {} Answer: {}.\"\n      prompt = \" \".join(\n          [\n              template.format(memory[i][0], memory[i][1])\n              for i in range(len(memory))\n          ]\n      ) + \" Question: \" + question + \" Answer:\"\n\n    # Generate text\n    inputs = blip_processor(image, text=prompt, return_tensors=\"pt\")\n    inputs = inputs.to(device, torch.float16)\n    generated_ids = model.generate(**inputs, max_new_tokens=100)\n    generated_text = blip_processor.batch_decode(\n        generated_ids,\n        skip_special_tokens=True\n    )\n    generated_text = generated_text[0].strip().split(\"Question\")[0]\n\n    # Update memory\n    memory.append((question, generated_text))\n\n    # Assign to output\n    output.append_display_data(HTML(\"<b>USER:</b> \" + question))\n    output.append_display_data(HTML(\"<b>BLIP-2:</b> \" + generated_text))\n    output.append_display_data(HTML(\"<br>\"))\n\n# Prepare widgets\nin_text = widgets.Text()\nin_text.continuous_update = False\nin_text.observe(text_eventhandler, \"value\")\noutput = widgets.Output()\nmemory = []\n\n# Display chat box\ndisplay(\n    widgets.VBox(\n        children=[output, in_text],\n        layout=widgets.Layout(display=\"inline-flex\", flex_flow=\"column-reverse\"),\n    )\n)",
    "crumbs": [
      "Home",
      "Applications",
      "9. Multimodal LLMs"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html",
    "href": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html",
    "title": "Chapter 7 - Advanced Text Generation Techniques and Tools",
    "section": "",
    "text": "!wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-fp16.gguf\n\n# If this command does not work for you, you can use the link directly to download the model\n# https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-fp16.gguf\n\n\n# from langchain import LlamaCpp\nfrom langchain_community.chat_models import ChatLlamaCpp\n\n\n\n# Make sure the model path is correct for your system!\nllm = ChatLlamaCpp(\n    model_path=\"Phi-3-mini-4k-instruct-fp16.gguf\",\n    n_gpu_layers=-1,\n    max_tokens=500,\n    n_ctx=2048,\n    seed=42,\n    verbose=False\n)\n\nllama_context: n_batch is less than GGML_KQ_MASK_PAD - increasing to 64\nllama_context: n_ctx_per_seq (2048) < n_ctx_train (4096) -- the full capacity of the model will not be utilized\n\n\n\nllm.invoke(\"Hi! My name is Maarten. What is 1 + 1?\")\n\nAIMessage(content=' Hello Maarten! The answer to 1 + 1 is 2.', additional_kwargs={}, response_metadata={'finish_reason': 'stop'}, id='lc_run--019bda17-2105-73c0-875e-d411cd445f4c-0', tool_calls=[], invalid_tool_calls=[])\n\n\n\n\n\nfrom langchain import PromptTemplate\n\n# Create a prompt template with the \"input_prompt\" variable\ntemplate = \"\"\"<s><|user|>\n{input_prompt}<|end|>\n<|assistant|>\"\"\"\nprompt = PromptTemplate(\n    template=template,\n    input_variables=[\"input_prompt\"]\n)\n\n\n---------------------------------------------------------------------------\nImportError                               Traceback (most recent call last)\nCell In[4], line 1\n----> 1 from langchain import PromptTemplate\n      3 # Create a prompt template with the \"input_prompt\" variable\n      4 template = \"\"\"<s><|user|>\n      5 {input_prompt}<|end|>\n      6 <|assistant|>\"\"\"\n\nImportError: cannot import name 'PromptTemplate' from 'langchain' (/home/alal/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/langchain/__init__.py)\n\n\n\n\nbasic_chain = prompt | llm\n\n\n# Use the chain\nbasic_chain.invoke(\n    {\n        \"input_prompt\": \"Hi! My name is Maarten. What is 1 + 1?\",\n    }\n)\n\n' Hello Maarten, the answer to 1 + 1 is 2.'\n\n\n\n\n\n\nfrom langchain import LLMChain\n\n# Create a chain for the title of our story\ntemplate = \"\"\"<s><|user|>\nCreate a title for a story about {summary}. Only return the title.<|end|>\n<|assistant|>\"\"\"\ntitle_prompt = PromptTemplate(template=template, input_variables=[\"summary\"])\ntitle = LLMChain(llm=llm, prompt=title_prompt, output_key=\"title\")\n\n/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:141: LangChainDeprecationWarning: The class `LLMChain` was deprecated in LangChain 0.1.17 and will be removed in 1.0. Use RunnableSequence, e.g., `prompt | llm` instead.\n  warn_deprecated(\n\n\n\ntitle.invoke({\"summary\": \"a girl that lost her mother\"})\n\n{'summary': 'a girl that lost her mother',\n 'title': ' \"Whispers of Love: A Journey Through Grief\"'}\n\n\n\n# Create a chain for the character description using the summary and title\ntemplate = \"\"\"<s><|user|>\nDescribe the main character of a story about {summary} with the title {title}. Use only two sentences.<|end|>\n<|assistant|>\"\"\"\ncharacter_prompt = PromptTemplate(\n    template=template, input_variables=[\"summary\", \"title\"]\n)\ncharacter = LLMChain(llm=llm, prompt=character_prompt, output_key=\"character\")\n\n\n# Create a chain for the story using the summary, title, and character description\ntemplate = \"\"\"<s><|user|>\nCreate a story about {summary} with the title {title}. The main charachter is: {character}. Only return the story and it cannot be longer than one paragraph<|end|>\n<|assistant|>\"\"\"\nstory_prompt = PromptTemplate(\n    template=template, input_variables=[\"summary\", \"title\", \"character\"]\n)\nstory = LLMChain(llm=llm, prompt=story_prompt, output_key=\"story\")\n\n\n# Combine all three components to create the full chain\nllm_chain = title | character | story\n\n\nllm_chain.invoke(\"a girl that lost her mother\")\n\n{'summary': 'a girl that lost her mother',\n 'title': ' \"In Loving Memory: A Journey Through Grief\"',\n 'character': ' The protagonist, Emily, is a resilient young girl who struggles to cope with her overwhelming grief after losing her beloved and caring mother at an early age. As she embarks on a journey of self-discovery and healing, she learns valuable life lessons from the memories and wisdom shared by those around her.',\n 'story': \" In Loving Memory: A Journey Through Grief revolves around Emily, a resilient young girl who loses her beloved mother at an early age. Struggling to cope with overwhelming grief, she embarks on a journey of self-discovery and healing, drawing strength from the cherished memories and wisdom shared by those around her. Through this transformative process, Emily learns valuable life lessons about resilience, love, and the power of human connection, ultimately finding solace in honoring her mother's legacy while embracing a newfound sense of inner peace amidst the painful loss.\"}",
    "crumbs": [
      "Home",
      "Applications",
      "7. Advanced generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationbuffer",
    "href": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationbuffer",
    "title": "Chapter 7 - Advanced Text Generation Techniques and Tools",
    "section": "ConversationBuffer",
    "text": "ConversationBuffer\n\n# Create an updated prompt template to include a chat history\ntemplate = \"\"\"<s><|user|>Current conversation:{chat_history}\n\n{input_prompt}<|end|>\n<|assistant|>\"\"\"\n\nprompt = PromptTemplate(\n    template=template,\n    input_variables=[\"input_prompt\", \"chat_history\"]\n)\n\n\nfrom langchain.memory import ConversationBufferMemory\n\n# Define the type of Memory we will use\nmemory = ConversationBufferMemory(memory_key=\"chat_history\")\n\n# Chain the LLM, Prompt, and Memory together\nllm_chain = LLMChain(\n    prompt=prompt,\n    llm=llm,\n    memory=memory\n)\n\n\n# Generate a conversation and ask a basic question\nllm_chain.invoke({\"input_prompt\": \"Hi! My name is Maarten. What is 1 + 1?\"})\n\n{'input_prompt': 'Hi! My name is Maarten. What is 1 + 1?',\n 'chat_history': '',\n 'text': \" Hello Maarten! The answer to 1 + 1 is 2. Hope you're having a great day!\"}\n\n\n\n# Does the LLM remember the name we gave it?\nllm_chain.invoke({\"input_prompt\": \"What is my name?\"})\n\n{'input_prompt': 'What is my name?',\n 'chat_history': \"Human: Hi! My name is Maarten. What is 1 + 1?\\nAI:  Hello Maarten! The answer to 1 + 1 is 2. Hope you're having a great day!\",\n 'text': ' Your name is Maarten.'}",
    "crumbs": [
      "Home",
      "Applications",
      "7. Advanced generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationbuffermemorywindow",
    "href": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationbuffermemorywindow",
    "title": "Chapter 7 - Advanced Text Generation Techniques and Tools",
    "section": "ConversationBufferMemoryWindow",
    "text": "ConversationBufferMemoryWindow\n\nfrom langchain.memory import ConversationBufferWindowMemory\n\n# Retain only the last 2 conversations in memory\nmemory = ConversationBufferWindowMemory(k=2, memory_key=\"chat_history\")\n\n# Chain the LLM, Prompt, and Memory together\nllm_chain = LLMChain(\n    prompt=prompt,\n    llm=llm,\n    memory=memory\n)\n\n\n# Ask two questions and generate two conversations in its memory\nllm_chain.invoke({\"input_prompt\":\"Hi! My name is Maarten and I am 33 years old. What is 1 + 1?\"})\nllm_chain.invoke({\"input_prompt\":\"What is 3 + 3?\"})\n\n{'input_prompt': 'What is 3 + 3?',\n 'chat_history': \"Human: Hi! My name is Maarten and I am 33 years old. What is 1 + 1?\\nAI:  Hello Maarten, it's nice to meet you! The answer to 1 + 1 is 2.\\n\\nHowever, if you have any other questions or need further assistance, feel free to ask!\",\n 'text': \" Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!\"}\n\n\n\n# Check whether it knows the name we gave it\nllm_chain.invoke({\"input_prompt\":\"What is my name?\"})\n\n{'input_prompt': 'What is my name?',\n 'chat_history': \"Human: Hi! My name is Maarten and I am 33 years old. What is 1 + 1?\\nAI:  Hello Maarten, it's nice to meet you! The answer to 1 + 1 is 2.\\n\\nHowever, if you have any other questions or need further assistance, feel free to ask!\\nHuman: What is 3 + 3?\\nAI:  Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!\",\n 'text': ' Your name is Maarten.'}\n\n\n\n# Check whether it knows the age we gave it\nllm_chain.invoke({\"input_prompt\":\"What is my age?\"})\n\n{'input_prompt': 'What is my age?',\n 'chat_history': \"Human: What is 3 + 3?\\nAI:  Hello again! 3 + 3 equals 6. If there's anything else I can help you with, just let me know!\\nHuman: What is my name?\\nAI:  Your name is Maarten.\",\n 'text': \" I'm unable to determine your age as I don't have access to personal information. Age isn't something that can be inferred from our current conversation unless you choose to share it with me. How else may I assist you today?\"}",
    "crumbs": [
      "Home",
      "Applications",
      "7. Advanced generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationsummary",
    "href": "Hands-On-Large-Language-Models/chapter07/Chapter 7 - Advanced Text Generation Techniques and Tools.html#conversationsummary",
    "title": "Chapter 7 - Advanced Text Generation Techniques and Tools",
    "section": "ConversationSummary",
    "text": "ConversationSummary\n\n# Create a summary prompt template\nsummary_prompt_template = \"\"\"<s><|user|>Summarize the conversations and update with the new lines.\n\nCurrent summary:\n{summary}\n\nnew lines of conversation:\n{new_lines}\n\nNew summary:<|end|>\n<|assistant|>\"\"\"\nsummary_prompt = PromptTemplate(\n    input_variables=[\"new_lines\", \"summary\"],\n    template=summary_prompt_template\n)\n\n\nfrom langchain.memory import ConversationSummaryMemory\n\n# Define the type of memory we will use\nmemory = ConversationSummaryMemory(\n    llm=llm,\n    memory_key=\"chat_history\",\n    prompt=summary_prompt\n)\n\n# Chain the LLM, prompt, and memory together\nllm_chain = LLMChain(\n    prompt=prompt,\n    llm=llm,\n    memory=memory\n)\n\n\n# Generate a conversation and ask for the name\nllm_chain.invoke({\"input_prompt\": \"Hi! My name is Maarten. What is 1 + 1?\"})\nllm_chain.invoke({\"input_prompt\": \"What is my name?\"})\n\n{'input_prompt': 'What is my name?',\n 'chat_history': ' Summary: Human, identified as Maarten, asked the AI about the sum of 1 + 1, which was correctly answered by the AI as 2 and offered additional assistance if needed.',\n 'text': ' Your name in this context was referred to as \"Maarten\". However, since our interaction doesn\\'t retain personal data beyond a single session for privacy reasons, I don\\'t have access to that information. How can I assist you further today?'}\n\n\n\n# Check whether it has summarized everything thus far\nllm_chain.invoke({\"input_prompt\": \"What was the first question I asked?\"})\n\n{'input_prompt': 'What was the first question I asked?',\n 'chat_history': ' Summary: Human, identified as Maarten in the context of this conversation, first asked about the sum of 1 + 1 and received an answer of 2 from the AI. Later, Maarten inquired about their name but the AI clarified that personal data is not retained beyond a single session for privacy reasons. The AI offered further assistance if needed.',\n 'text': ' The first question you asked was \"what\\'s 1 + 1?\"'}\n\n\n\n# Check what the summary is thus far\nmemory.load_memory_variables({})\n\n{'chat_history': ' Maarten, identified in this conversation, initially asked about the sum of 1+1 which resulted in an answer from the AI being 2. Subsequently, he sought clarification on his name but the AI informed him that no personal data is retained beyond a single session due to privacy reasons. The AI then offered further assistance if required. Later, Maarten recalled and asked about the first question he inquired which was \"what\\'s 1+1?\"'}",
    "crumbs": [
      "Home",
      "Applications",
      "7. Advanced generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "",
    "text": "# Load data from huggingface\nfrom datasets import load_dataset\ndataset = load_dataset(\"maartengr/arxiv_nlp\")[\"train\"]\n\n# Extract metadata\nabstracts = list(dataset[\"Abstracts\"])\ntitles = list(dataset[\"Titles\"])",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#embedding-documents",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#embedding-documents",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "1. Embedding Documents",
    "text": "1. Embedding Documents\n\nfrom sentence_transformers import SentenceTransformer\n\n# Create an embedding for each abstract\nembedding_model = SentenceTransformer('thenlper/gte-small')\nembeddings = embedding_model.encode(abstracts, show_progress_bar=True)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Check the dimensions of the resulting embeddings\nembeddings.shape\n\n(44949, 384)",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#reducing-the-dimensionality-of-embeddings",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#reducing-the-dimensionality-of-embeddings",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "2. Reducing the Dimensionality of Embeddings",
    "text": "2. Reducing the Dimensionality of Embeddings\n\nfrom umap import UMAP\n\n# We reduce the input embeddings from 384 dimenions to 5 dimenions\numap_model = UMAP(\n    n_components=5, min_dist=0.0, metric='cosine', random_state=42\n)\nreduced_embeddings = umap_model.fit_transform(embeddings)\n\n/home/alal/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.\n  warn(",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#cluster-the-reduced-embeddings",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#cluster-the-reduced-embeddings",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "3. Cluster the Reduced Embeddings",
    "text": "3. Cluster the Reduced Embeddings\n\nfrom hdbscan import HDBSCAN\n\n\n# We fit the model and extract the clusters\nhdbscan_model = HDBSCAN(\n    min_cluster_size=50, metric='euclidean', cluster_selection_method='eom'\n).fit(reduced_embeddings)\nclusters = hdbscan_model.labels_\n\n# How many clusters did we generate?\nlen(set(clusters))\n\n164",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#inspecting-the-clusters",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#inspecting-the-clusters",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "Inspecting the Clusters",
    "text": "Inspecting the Clusters\nManually inspect the first three documents in cluster 0:\n\nimport numpy as np\n\n# Print first three documents in cluster 0\ncluster = 0\nfor index in np.where(clusters==cluster)[0][:3]:\n    print(abstracts[index][:300] + \"... \\n\")\n\n  This works aims to design a statistical machine translation from English text\nto American Sign Language (ASL). The system is based on Moses tool with some\nmodifications and the results are synthesized through a 3D avatar for\ninterpretation. First, we translate the input text to gloss, a written fo... \n\n  Researches on signed languages still strongly dissociate lin- guistic issues\nrelated on phonological and phonetic aspects, and gesture studies for\nrecognition and synthesis purposes. This paper focuses on the imbrication of\nmotion and meaning for the analysis, synthesis and evaluation of sign lang... \n\n  Modern computational linguistic software cannot produce important aspects of\nsign language translation. Using some researches we deduce that the majority of\nautomatic sign language translation systems ignore many aspects when they\ngenerate animation; therefore the interpretation lost the truth inf... \n\n\n\nNext, we reduce our embeddings to 2-dimensions so that we can plot them and get a rough understanding of the generated clusters.\n\nimport pandas as pd\n\n# Reduce 384-dimensional embeddings to 2 dimensions for easier visualization\nreduced_embeddings = UMAP(\n    n_components=2, min_dist=0.0, metric='cosine', random_state=42\n).fit_transform(embeddings)\n\n# Create dataframe\ndf = pd.DataFrame(reduced_embeddings, columns=[\"x\", \"y\"])\ndf[\"title\"] = titles\ndf[\"cluster\"] = [str(c) for c in clusters]\n\n# Select outliers and non-outliers (clusters)\nclusters_df = df.loc[df.cluster != \"-1\", :]\noutliers_df = df.loc[df.cluster == \"-1\", :]\n\n/home/alal/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.\n  warn(\n\n\n\nStatic Plot\n\nimport matplotlib.pyplot as plt\n\n# Plot outliers and non-outliers seperately\nplt.scatter(outliers_df.x, outliers_df.y, alpha=0.05, s=2, c=\"grey\")\nplt.scatter(\n    clusters_df.x, clusters_df.y, c=clusters_df.cluster.astype(int),\n    alpha=0.6, s=2, cmap='tab20b'\n)\nplt.axis('off')\n# plt.savefig(\"matplotlib.png\", dpi=300)  # Uncomment to save the graph as a .png\n\n(np.float64(-7.382833409309387),\n np.float64(10.984574723243714),\n np.float64(-2.5725981533527373),\n np.float64(16.283057743310927))",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#bertopic-a-modular-topic-modeling-framework",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#bertopic-a-modular-topic-modeling-framework",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "BERTopic: A Modular Topic Modeling Framework",
    "text": "BERTopic: A Modular Topic Modeling Framework\n\nfrom bertopic import BERTopic\n\n# Train our model with our previously defined models\ntopic_model = BERTopic(\n    embedding_model=embedding_model,\n    umap_model=umap_model,\n    hdbscan_model=hdbscan_model,\n    verbose=True\n).fit(abstracts, embeddings)\n\n2026-01-19 21:47:57,272 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm\n2026-01-19 21:48:21,113 - BERTopic - Dimensionality - Completed ✓\n2026-01-19 21:48:21,114 - BERTopic - Cluster - Start clustering the reduced embeddings\n2026-01-19 21:48:21,828 - BERTopic - Cluster - Completed ✓\n2026-01-19 21:48:21,833 - BERTopic - Representation - Fine-tuning topics using representation models.\n2026-01-19 21:48:23,635 - BERTopic - Representation - Completed ✓\n\n\nNow, let’s start exploring the topics that we got by running the code above.\n\ntopic_model.get_topic_info()\n\n\n\n\n\n\n\n\nTopic\nCount\nName\nRepresentation\nRepresentative_Docs\n\n\n\n\n0\n-1\n15711\n-1_the_of_and_to\n[the, of, and, to, in, we, for, language, that...\n[ Few-shot question answering (QA) aims at ac...\n\n\n1\n0\n2055\n0_speech_asr_recognition_end\n[speech, asr, recognition, end, acoustic, audi...\n[ All-neural, end-to-end ASR systems gained r...\n\n\n2\n1\n1089\n1_translation_nmt_machine_bleu\n[translation, nmt, machine, bleu, neural, engl...\n[ Neural machine translation (NMT) becomes a ...\n\n\n3\n2\n853\n2_summarization_summaries_summary_abstractive\n[summarization, summaries, summary, abstractiv...\n[ Evaluation of text summarization approaches...\n\n\n4\n3\n829\n3_hate_offensive_speech_detection\n[hate, offensive, speech, detection, toxic, so...\n[ The goal of hate speech detection is to fil...\n\n\n...\n...\n...\n...\n...\n...\n\n\n159\n158\n53\n158_spelling_csc_correction_chinese\n[spelling, csc, correction, chinese, errors, e...\n[ Chinese Spell Checking (CSC) task aims to d...\n\n\n160\n159\n53\n159_counseling_mental_therapy_health\n[counseling, mental, therapy, health, psychoth...\n[ Mental health care poses an increasingly se...\n\n\n161\n160\n52\n160_reviews_opinion_summaries_summarization\n[reviews, opinion, summaries, summarization, r...\n[ When faced with a large number of product r...\n\n\n162\n161\n52\n161_gans_gan_adversarial_generation\n[gans, gan, adversarial, generation, generativ...\n[ Text generation is of particular interest i...\n\n\n163\n162\n52\n162_backdoor_attacks_attack_triggers\n[backdoor, attacks, attack, triggers, poisoned...\n[ Deep neural networks (DNNs) and natural lan...\n\n\n\n\n164 rows × 5 columns\n\n\n\nHundreds of topics were generated using the default model! To get the top 10 keywords per topic as well as their c-TF-IDF weights, we can use the get_topic() function:\n\ntopic_model.get_topic(0)\n\n[('speech', np.float64(0.028044048463799758)),\n ('asr', np.float64(0.019103557225532256)),\n ('recognition', np.float64(0.013552091477301144)),\n ('end', np.float64(0.010289302278892161)),\n ('acoustic', np.float64(0.00973514556960071)),\n ('audio', np.float64(0.006846988836039268)),\n ('speaker', np.float64(0.006746370403108714)),\n ('wer', np.float64(0.0065942814733770585)),\n ('error', np.float64(0.006503600337995727)),\n ('automatic', np.float64(0.006073033426036759))]\n\n\nWe can use the find_topics() function to search for specific topics based on a search term. Let’s search for a topic about topic modeling:\n\ntopic_model.find_topics(\"topic modeling\")\n\n([25, -1, 43, 39, 88],\n [np.float32(0.9548093),\n  np.float32(0.9129914),\n  np.float32(0.9078753),\n  np.float32(0.90433663),\n  np.float32(0.9037678)])\n\n\nIt returns that topic 22 has a relatively high similarity (0.95) with our search term. If we then inspect the topic, we can see that it is indeed a topic about topic modeling:\n\ntopic_model.get_topic(25)\n\n[('topic', np.float64(0.06762386852487184)),\n ('topics', np.float64(0.035761575158132886)),\n ('lda', np.float64(0.016942399804845947)),\n ('latent', np.float64(0.013364966654403065)),\n ('documents', np.float64(0.012795304337419438)),\n ('document', np.float64(0.012745519708000716)),\n ('modeling', np.float64(0.012114985329208914)),\n ('dirichlet', np.float64(0.010212517055996743)),\n ('word', np.float64(0.008675659463331778)),\n ('allocation', np.float64(0.007980208492860665))]\n\n\nThat seems like a topic that is, in part, characterized by the classic LDA technique. Let’s see if the BERTopic paper was also assigned to topic 22:\n\ntopic_model.topics_[titles.index('BERTopic: Neural topic modeling with a class-based TF-IDF procedure')]\n\n25\n\n\nIt is! We expected it might be because there are non-LDA specific words in the topic describtion such as “clustering” and “topic”.\n\nVisualizations\nVisualize Documents\n\n# Visualize topics and documents\nfig = topic_model.visualize_documents(\n    titles,\n    reduced_embeddings=reduced_embeddings,\n    width=1200,\n    hide_annotations=True\n)\n\n# Update fonts of legend for easier visualization\nfig.update_layout(font=dict(size=16))\n\n\n---------------------------------------------------------------------------\nValueError                                Traceback (most recent call last)\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/IPython/core/formatters.py:984, in IPythonDisplayFormatter.__call__(self, obj)\n    982 method = get_real_method(obj, self.print_method)\n    983 if method is not None:\n--> 984     method()\n    985     return True\n\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/plotly/basedatatypes.py:850, in BaseFigure._ipython_display_(self)\n    847 import plotly.io as pio\n    849 if pio.renderers.render_on_display and pio.renderers.default:\n--> 850     pio.show(self)\n    851 else:\n    852     print(repr(self))\n\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/plotly/io/_renderers.py:415, in show(fig, renderer, validate, **kwargs)\n    410     raise ValueError(\n    411         \"Mime type rendering requires ipython but it is not installed\"\n    412     )\n    414 if not nbformat or Version(nbformat.__version__) < Version(\"4.2.0\"):\n--> 415     raise ValueError(\n    416         \"Mime type rendering requires nbformat>=4.2.0 but it is not installed\"\n    417     )\n    419 display_jupyter_version_warnings()\n    421 ipython_display.display(bundle, raw=True)\n\nValueError: Mime type rendering requires nbformat>=4.2.0 but it is not installed\n\n\n\n                        \n                                            \n\n\n\n# Visualize barchart with ranked keywords\ntopic_model.visualize_barchart()\n\n# Visualize relationships between topics\ntopic_model.visualize_heatmap(n_clusters=30)\n\n# Visualize the potential hierarchical structure of topics\ntopic_model.visualize_hierarchy()\n\n\n---------------------------------------------------------------------------\nValueError                                Traceback (most recent call last)\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/IPython/core/formatters.py:984, in IPythonDisplayFormatter.__call__(self, obj)\n    982 method = get_real_method(obj, self.print_method)\n    983 if method is not None:\n--> 984     method()\n    985     return True\n\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/plotly/basedatatypes.py:850, in BaseFigure._ipython_display_(self)\n    847 import plotly.io as pio\n    849 if pio.renderers.render_on_display and pio.renderers.default:\n--> 850     pio.show(self)\n    851 else:\n    852     print(repr(self))\n\nFile ~/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/plotly/io/_renderers.py:415, in show(fig, renderer, validate, **kwargs)\n    410     raise ValueError(\n    411         \"Mime type rendering requires ipython but it is not installed\"\n    412     )\n    414 if not nbformat or Version(nbformat.__version__) < Version(\"4.2.0\"):\n--> 415     raise ValueError(\n    416         \"Mime type rendering requires nbformat>=4.2.0 but it is not installed\"\n    417     )\n    419 display_jupyter_version_warnings()\n    421 ipython_display.display(bundle, raw=True)\n\nValueError: Mime type rendering requires nbformat>=4.2.0 but it is not installed",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#representation-models",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#representation-models",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "Representation Models",
    "text": "Representation Models\nIn these examples that follow, we will update our topic representations after having trained our model. This allows for quick iteration. If, however, you want to use a representation model at the start of training, you will need to run it as follows:\nfrom bertopic.representation import KeyBERTInspired\nfrom bertopic import BERTopic\n\n# Create your representation model\nrepresentation_model = KeyBERTInspired()\n\n# Use the representation model in BERTopic on top of the default pipeline\ntopic_model = BERTopic(representation_model=representation_model)\nTo use the representation models, we are first going to duplicate our topic model such that easily show the differences between a model with and without representation model.\n\n# Save original representations\nfrom copy import deepcopy\noriginal_topics = deepcopy(topic_model.topic_representations_)\n\n\ndef topic_differences(model, original_topics, nr_topics=5):\n    \"\"\"Show the differences in topic representations between two models \"\"\"\n    df = pd.DataFrame(columns=[\"Topic\", \"Original\", \"Updated\"])\n    for topic in range(nr_topics):\n\n        # Extract top 5 words per topic per model\n        og_words = \" | \".join(list(zip(*original_topics[topic]))[0][:5])\n        new_words = \" | \".join(list(zip(*model.get_topic(topic)))[0][:5])\n        df.loc[len(df)] = [topic, og_words, new_words]\n\n    return df\n\n\nKeyBERTInspired\n\nfrom bertopic.representation import KeyBERTInspired\n\n# Update our topic representations using KeyBERTInspired\nrepresentation_model = KeyBERTInspired()\ntopic_model.update_topics(abstracts, representation_model=representation_model)\n\n# Show topic differences\ntopic_differences(topic_model, original_topics)\n\n\n\n\n\n\n\n\nTopic\nOriginal\nUpdated\n\n\n\n\n0\n0\nspeech | asr | recognition | end | acoustic\nlanguage | phonetic | speech | encoder | trans...\n\n\n1\n1\ntranslation | nmt | machine | bleu | neural\ntranslation | translate | translations | trans...\n\n\n2\n2\nsummarization | summaries | summary | abstract...\nsummarization | summarizers | summaries | summ...\n\n\n3\n3\nhate | offensive | speech | detection | toxic\nhate | hateful | language | languages | offensive\n\n\n4\n4\nrelation | extraction | re | relations | entity\nrelation | relations | relational | extracting...\n\n\n\n\n\n\n\n\n\nMaximal Marginal Relevance\n\nfrom bertopic.representation import MaximalMarginalRelevance\n\n# Update our topic representations to MaximalMarginalRelevance\nrepresentation_model = MaximalMarginalRelevance(diversity=0.5)\ntopic_model.update_topics(abstracts, representation_model=representation_model)\n\n# Show topic differences\ntopic_differences(topic_model, original_topics)\n\n\n\n\n\n\n\n\nTopic\nOriginal\nUpdated\n\n\n\n\n0\n0\nspeech | asr | recognition | end | acoustic\nspeech | asr | error | model | training\n\n\n1\n1\ntranslation | nmt | machine | bleu | neural\ntranslation | nmt | bleu | parallel | multilin...\n\n\n2\n2\nsummarization | summaries | summary | abstract...\nsummarization | extractive | rouge | sentences...\n\n\n3\n3\nhate | offensive | speech | detection | toxic\noffensive | toxic | hateful | platforms | dataset\n\n\n4\n4\nrelation | extraction | re | relations | entity\nextraction | re | relations | entity | level",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#text-generation",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#text-generation",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "Text Generation",
    "text": "Text Generation\n\nFlan-T5\n\nfrom transformers import pipeline\nfrom bertopic.representation import TextGeneration\n\nprompt = \"\"\"I have a topic that contains the following documents:\n[DOCUMENTS]\n\nThe topic is described by the following keywords: '[KEYWORDS]'.\n\nBased on the documents and keywords, what is this topic about?\"\"\"\n\n# Update our topic representations using Flan-T5\ngenerator = pipeline('text2text-generation', model='google/flan-t5-small')\nrepresentation_model = TextGeneration(\n    generator, prompt=prompt, doc_length=50, tokenizer=\"whitespace\"\n)\ntopic_model.update_topics(abstracts, representation_model=representation_model)\n\n# Show topic differences\ntopic_differences(topic_model, original_topics)\n\nDevice set to use cuda:0\n  6%|▌         | 10/164 [00:00<00:10, 14.76it/s]You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n100%|██████████| 164/164 [00:18<00:00,  8.74it/s]\n\n\n\n\n\n\n\n\n\nTopic\nOriginal\nUpdated\n\n\n\n\n0\n0\nspeech | asr | recognition | end | acoustic\nSpeech-to-sequence modeling | | | |\n\n\n1\n1\ntranslation | nmt | machine | bleu | neural\nScience/Tech | | | |\n\n\n2\n2\nsummarization | summaries | summary | abstract...\nSummarization | | | |\n\n\n3\n3\nhate | offensive | speech | detection | toxic\nhate speech | | | |\n\n\n4\n4\nrelation | extraction | re | relations | entity\nrelation extraction | | | |\n\n\n\n\n\n\n\n\n\nOpenAI\n\nimport openai\nfrom bertopic.representation import OpenAI\n\nprompt = \"\"\"\nI have a topic that contains the following documents:\n[DOCUMENTS]\n\nThe topic is described by the following keywords: [KEYWORDS]\n\nBased on the information above, extract a short topic label in the following format:\ntopic: <short topic label>\n\"\"\"\n\n# Update our topic representations using GPT-3.5\nclient = openai.OpenAI(api_key=\"YOUR_KEY_HERE\")\nrepresentation_model = OpenAI(\n    client, model=\"gpt-3.5-turbo\", exponential_backoff=True, chat=True, prompt=prompt\n)\ntopic_model.update_topics(abstracts, representation_model=representation_model)\n\n# Show topic differences\ntopic_differences(topic_model, original_topics)\n\n100%|██████████| 156/156 [02:13<00:00,  1.17it/s]\n\n\n   Topic                                           Original  \\\n0      0        speech | asr | recognition | end | acoustic   \n1      1  medical | clinical | biomedical | patient | he...   \n2      2  sentiment | aspect | analysis | reviews | opinion   \n3      3        translation | nmt | machine | neural | bleu   \n4      4  summarization | summaries | summary | abstract...   \n\n                                             Updated  \n0  Leveraging External Data for Improving Low-Res...  \n1  Improved Representation Learning for Biomedica...  \n2  \"Advancements in Aspect-Based Sentiment Analys...  \n3            Neural Machine Translation Enhancements  \n4                  Document Summarization Techniques  \n\n\n\n# Visualize topics and documents\nfig = topic_model.visualize_document_datamap(\n    titles,\n    topics=list(range(20)),\n    reduced_embeddings=reduced_embeddings,\n    width=1200,\n    label_font_size=11,\n    label_wrap_width=20,\n    use_medoids=True,\n)\nplt.savefig(\"datamapplot.png\", dpi=300)",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#bonus-word-cloud",
    "href": "Hands-On-Large-Language-Models/chapter05/Chapter 5 - Text Clustering and Topic Modeling.html#bonus-word-cloud",
    "title": "Chapter 5 - Text Clustering and Topic Modeling",
    "section": "BONUS: Word Cloud",
    "text": "BONUS: Word Cloud\nMake sure to pip install wordcloud first in order to follow this bonus:\nFirst, we need to make sure that each topic is described by a bit more words than just 10 as that would make for a much more interesting wordcloud.\n\ntopic_model.update_topics(abstracts, top_n_words=500)\n\nThen, we can run the following code to generate the wordcloud for our topic modeling topic:\n\nfrom wordcloud import WordCloud\nimport matplotlib.pyplot as plt\n\ndef create_wordcloud(model, topic):\n    plt.figure(figsize=(10,5))\n    text = {word: value for word, value in model.get_topic(topic)}\n    wc = WordCloud(background_color=\"white\", max_words=1000, width=1600, height=800)\n    wc.generate_from_frequencies(text)\n    plt.imshow(wc, interpolation=\"bilinear\")\n    plt.axis(\"off\")\n    plt.show()\n\n# Show wordcloud\ncreate_wordcloud(topic_model, topic=17)",
    "crumbs": [
      "Home",
      "Applications",
      "5. Clustering and topics"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter03/Chapter 3 - Looking Inside LLMs.html",
    "href": "Hands-On-Large-Language-Models/chapter03/Chapter 3 - Looking Inside LLMs.html",
    "title": "Chapter 3 - Looking Inside LLMs",
    "section": "",
    "text": "Loading the LLM\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n\n\n# Load model and tokenizer\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"microsoft/Phi-3-mini-4k-instruct\",\n    device_map=\"cuda\",\n    torch_dtype=\"auto\",\n    trust_remote_code=False,\n)\n\n# Create a pipeline\ngenerator = pipeline(\n    \"text-generation\",\n    model=model,\n    tokenizer=tokenizer,\n    return_full_text=False,\n    max_new_tokens=50,\n    do_sample=False,\n)\n\n`torch_dtype` is deprecated! Use `dtype` instead!\n\n\n\n\n\nDevice set to use cuda\nThe following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n\n\n\n\nThe Inputs and Outputs of a Trained Transformer LLM\n\nprompt = \"Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.\"\n\noutput = generator(prompt)\n\nprint(output[0]['generated_text'])\n\n Mention that you've already taken steps to prevent it in the future.\n\n\nEmail to Sarah:\n\nSubject: Sincere Apologies for the Gardening Mishap\n\n\nDear Sarah,\n\n\nI\n\n\n\nprint(model)\n\nPhi3ForCausalLM(\n  (model): Phi3Model(\n    (embed_tokens): Embedding(32064, 3072, padding_idx=32000)\n    (layers): ModuleList(\n      (0-31): 32 x Phi3DecoderLayer(\n        (self_attn): Phi3Attention(\n          (o_proj): Linear(in_features=3072, out_features=3072, bias=False)\n          (qkv_proj): Linear(in_features=3072, out_features=9216, bias=False)\n        )\n        (mlp): Phi3MLP(\n          (gate_up_proj): Linear(in_features=3072, out_features=16384, bias=False)\n          (down_proj): Linear(in_features=8192, out_features=3072, bias=False)\n          (activation_fn): SiLUActivation()\n        )\n        (input_layernorm): Phi3RMSNorm((3072,), eps=1e-05)\n        (post_attention_layernorm): Phi3RMSNorm((3072,), eps=1e-05)\n        (resid_attn_dropout): Dropout(p=0.0, inplace=False)\n        (resid_mlp_dropout): Dropout(p=0.0, inplace=False)\n      )\n    )\n    (norm): Phi3RMSNorm((3072,), eps=1e-05)\n    (rotary_emb): Phi3RotaryEmbedding()\n  )\n  (lm_head): Linear(in_features=3072, out_features=32064, bias=False)\n)\n\n\n\n\nChoosing a single token from the probability distribution (sampling / decoding)\n\nprompt = \"The capital of France is\"\n\n# Tokenize the input prompt\ninput_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n# Tokenize the input prompt\ninput_ids = input_ids.to(\"cuda\")\n\n# Get the output of the model before the lm_head\nmodel_output = model.model(input_ids)\n\n# Get the output of the lm_head\nlm_head_output = model.lm_head(model_output[0])\n\n\ntoken_id = lm_head_output[0,-1].argmax(-1)\ntokenizer.decode(token_id)\n\n'Paris'\n\n\n\nmodel_output[0].shape\n\ntorch.Size([1, 5, 3072])\n\n\n\nlm_head_output.shape\n\ntorch.Size([1, 5, 32064])\n\n\n\n\nSpeeding up generation by caching keys and values\n\nprompt = \"Write a very long email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.\"\n\n# Tokenize the input prompt\ninput_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\ninput_ids = input_ids.to(\"cuda\")\n\n\n%%timeit -n 1\n# Generate the text\ngeneration_output = model.generate(\n  input_ids=input_ids,\n  max_new_tokens=100,\n  use_cache=True\n)\n\nThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n\n\n1.65 s ± 4.09 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n\n\n\n%%timeit -n 1\n# Generate the text\ngeneration_output = model.generate(\n  input_ids=input_ids,\n  max_new_tokens=100,\n  use_cache=False\n)\n\n3 s ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n\n\n\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Foundations",
      "3. Inside LLMs"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter01/Chapter 1 - Introduction to Language Models.html",
    "href": "Hands-On-Large-Language-Models/chapter01/Chapter 1 - Introduction to Language Models.html",
    "title": "Chapter 1 - Introduction to Language Models",
    "section": "",
    "text": "Phi-3\nThe first step is to load our model onto the GPU for faster inference. Note that we load the model and tokenizer separately (although that isn’t always necessary).\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"microsoft/Phi-3-mini-4k-instruct\",\n    device_map=\"cuda\",\n    torch_dtype=\"auto\",\n    trust_remote_code=False,\n    attn_implementation=\"flash_attention_2\",\n)\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n`torch_dtype` is deprecated! Use `dtype` instead!\n\n\n\n\n\nAlthough we can now use the model and tokenizer directly, it’s much easier to wrap it in a pipeline object:\n\nfrom transformers import pipeline\n\n# Create a pipeline\ngenerator = pipeline(\n    \"text-generation\",\n    model=model,\n    tokenizer=tokenizer,\n    return_full_text=False,\n    max_new_tokens=1000,\n    do_sample=False\n)\n\nDevice set to use cuda\nThe following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n\n\nFinally, we create our prompt as a user and give it to the model:\n\n# The prompt (user input / query)\nmessages = [\n    {\"role\": \"user\", \"content\": \"Write a funny joke about herons.\"} ]\n\n# Generate output\noutput = generator(messages)\nprint(output[0][\"generated_text\"])\n\n Why don't herons ever play hide and seek? Because good luck hiding when you're always standing out with your long legs and neck!\n\n\n\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Foundations",
      "1. Introduction"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter02/Chapter 2 - Tokens and Token Embeddings.html",
    "href": "Hands-On-Large-Language-Models/chapter02/Chapter 2 - Tokens and Token Embeddings.html",
    "title": "Chapter 2 - Tokens and Token Embeddings",
    "section": "",
    "text": "Downloading and Running An LLM\nThe first step is to load our model onto the GPU for faster inference. Note that we load the model and tokenizer separately and keep them as such so that we can explore them separately.\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"microsoft/Phi-3-mini-4k-instruct\",\n    device_map=\"cuda\",\n    torch_dtype=\"auto\",\n    trust_remote_code=False,\n    attn_implementation=\"flash_attention_2\",\n)\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n`torch_dtype` is deprecated! Use `dtype` instead!\n\n\n\n\n\n\nprompt = \"Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>\"\n\n# Tokenize the input prompt\ninput_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(\"cuda\")\n\n# Generate the text\ngeneration_output = model.generate(\n  input_ids=input_ids,\n  max_new_tokens=50,\n)\n\n# Print the output\nprint(tokenizer.decode(generation_output[0]))\n\nThe attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n\n\nWrite an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|> Subject: Sincere Apologies for the Gardening Mishap\n\n\nDear Sarah,\n\n\nI hope this message finds you well. I am writing to express my deepest apologies for the unfortunate incident that\n\n\n\nprint(input_ids)\n\ntensor([[14350,   385,  4876, 27746,  5281,   304, 19235,   363,   278, 25305,\n           293, 16423,   292,   286,   728,   481, 29889, 12027,  7420,   920,\n           372,  9559, 29889, 32001]], device='cuda:0')\n\n\n\nfor id in input_ids[0]:\n   print(tokenizer.decode(id))\n\nWrite\nan\nemail\napolog\nizing\nto\nSarah\nfor\nthe\ntrag\nic\ngarden\ning\nm\nish\nap\n.\nExp\nlain\nhow\nit\nhappened\n.\n<|assistant|>\n\n\n\ngeneration_output\n\ntensor([[14350,   385,  4876, 27746,  5281,   304, 19235,   363,   278, 25305,\n           293, 16423,   292,   286,   728,   481, 29889, 12027,  7420,   920,\n           372,  9559, 29889, 32001,  3323,   622, 29901,   317,  3742,   406,\n          6225, 11763,   363,   278, 19906,   292,   341,   728,   481,    13,\n            13,    13, 29928,   799, 19235, 29892,    13,    13,    13, 29902,\n          4966,   445,  2643, 14061,   366,  1532, 29889,   306,   626,  5007,\n           304,  4653,   590,  6483,   342,  3095, 11763,   363,   278,   443,\n          6477,   403, 15134,   393]], device='cuda:0')\n\n\n\nprint(tokenizer.decode(3323))\nprint(tokenizer.decode(622))\nprint(tokenizer.decode([3323, 622]))\nprint(tokenizer.decode(29901))\n\nSub\nject\nSubject\n:\n\n\n\n\nComparing Trained LLM Tokenizers\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncolors_list = [\n    '102;194;165', '252;141;98', '141;160;203',\n    '231;138;195', '166;216;84', '255;217;47'\n]\n\ndef show_tokens(sentence, tokenizer_name):\n    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n    token_ids = tokenizer(sentence).input_ids\n    for idx, t in enumerate(token_ids):\n        print(\n            f'\\x1b[0;30;48;2;{colors_list[idx % len(colors_list)]}m' +\n            tokenizer.decode(t) +\n            '\\x1b[0m',\n            end=' '\n        )\n\n\ntext = \"\"\"\nEnglish and CAPITALIZATION\n🎵 鸟\nshow_tokens False None elif == >= else: two tabs:\"    \" Three tabs: \"       \"\n12.0*50=600\n\"\"\"\n\n\nshow_tokens(text, \"bert-base-uncased\")\n\n[CLS] english and capital ##ization [UNK] [UNK] show _ token ##s false none eli ##f = = > = else : two tab ##s : \" \" three tab ##s : \" \" 12 . 0 * 50 = 600 [SEP] \n\n\n\nshow_tokens(text, \"bert-base-cased\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n[CLS] English and CA ##PI ##TA ##L ##I ##Z ##AT ##ION [UNK] [UNK] show _ token ##s F ##als ##e None el ##if = = > = else : two ta ##bs : \" \" Three ta ##bs : \" \" 12 . 0 * 50 = 600 [SEP] \n\n\n\nshow_tokens(text, \"gpt2\")\n\n\n English  and  CAP ITAL IZ ATION \n � � �  � � � \n show _ t ok ens  False  None  el if  ==  >=  else :  two  tabs :\"        \"  Three  tabs :  \"              \" \n 12 . 0 * 50 = 600 \n \n\n\n\nshow_tokens(text, \"google/flan-t5-small\")\n\nEnglish and CA PI TAL IZ ATION  <unk>  <unk> show _ to ken s Fal s e None  e l if = = > = else : two tab s : \" \" Three tab s : \" \" 12. 0 * 50 = 600  </s> \n\n\n\n# The official is `tiktoken` but this the same tokenizer on the HF platform\nshow_tokens(text, \"Xenova/gpt-4\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n English  and  CAPITAL IZATION \n � � �  � � � \n show _tokens  False  None  elif  ==  >=  else :  two  tabs :\"      \"  Three  tabs :  \"         \"\n 12 . 0 * 50 = 600 \n \n\n\n\n# You need to request access before being able to use this tokenizer\nshow_tokens(text, \"bigcode/starcoder2-15b\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n English  and  CAPITAL IZATION \n � � �   � � \n show _ tokens  False  None  elif  ==  >=  else :  two  tabs :\"      \"  Three  tabs :  \"         \" \n 1 2 . 0 * 5 0 = 6 0 0 \n \n\n\n\nshow_tokens(text, \"facebook/galactica-1.3b\")\n\n\n\n\n\n\n\n\n\n\n\n English  and  CAP ITAL IZATION \n � � � �  � � � \n show _ tokens  False  None  elif   ==   > =  else :  two  t abs : \"      \"  Three  t abs :   \"         \" \n 1 2 . 0 * 5 0 = 6 0 0 \n \n\n\n\nshow_tokens(text, \"microsoft/Phi-3-mini-4k-instruct\")\n\n \n English and C AP IT AL IZ ATION \n � � � �  � � � \n show _ to kens False None elif == >= else : two tabs :\"    \" Three tabs : \"       \" \n 1 2 . 0 * 5 0 = 6 0 0 \n \n\n\n\n\nContextualized Word Embeddings From a Language Model (Like BERT)\n\nfrom transformers import AutoModel, AutoTokenizer\n\n# Load a tokenizer\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/deberta-base\")\n\n# Load a language model\nmodel = AutoModel.from_pretrained(\"microsoft/deberta-v3-xsmall\")\n\n# Tokenize the sentence\ntokens = tokenizer('Hello world', return_tensors='pt')\n\n# Process the tokens\noutput = model(**tokens)[0]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\noutput.shape\n\ntorch.Size([1, 4, 384])\n\n\n\nfor token in tokens['input_ids'][0]:\n    print(tokenizer.decode(token))\n\n[CLS]\nHello\n world\n[SEP]\n\n\n\noutput\n\ntensor([[[-3.4816,  0.0861, -0.1819,  ..., -0.0612, -0.3911,  0.3017],\n         [ 0.1898,  0.3208, -0.2315,  ...,  0.3714,  0.2478,  0.8048],\n         [ 0.2071,  0.5036, -0.0485,  ...,  1.2175, -0.2292,  0.8582],\n         [-3.4278,  0.0645, -0.1427,  ...,  0.0658, -0.4367,  0.3834]]],\n       grad_fn=<NativeLayerNormBackward0>)\n\n\n\n\nText Embeddings (For Sentences and Whole Documents)\n\nfrom sentence_transformers import SentenceTransformer\n\n# Load model\nmodel = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')\n\n# Convert text to text embeddings\nvector = model.encode(\"Best movie ever!\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nvector.shape\n\n(768,)\n\n\n\n\nWord Embeddings Beyond LLMs\n\nimport gensim.downloader as api\n\n# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)\n# Other options include \"word2vec-google-news-300\"\n# More options at https://github.com/RaRe-Technologies/gensim-data\nmodel = api.load(\"glove-wiki-gigaword-50\")\n\n[==================================================] 100.0% 66.0/66.0MB downloaded\n\n\n\nmodel.most_similar([model['king']], topn=11)\n\n[('king', 1.0000001192092896),\n ('prince', 0.8236179351806641),\n ('queen', 0.7839043140411377),\n ('ii', 0.7746230363845825),\n ('emperor', 0.7736247777938843),\n ('son', 0.766719400882721),\n ('uncle', 0.7627150416374207),\n ('kingdom', 0.7542161345481873),\n ('throne', 0.7539914846420288),\n ('brother', 0.7492411136627197),\n ('ruler', 0.7434253692626953)]\n\n\n\nmodel.most_similar([model['queen']], topn=11)\n\n[('queen', 1.0000001192092896),\n ('princess', 0.8515166640281677),\n ('lady', 0.8050609230995178),\n ('elizabeth', 0.7873042225837708),\n ('king', 0.7839043140411377),\n ('prince', 0.7821861505508423),\n ('coronation', 0.769277811050415),\n ('consort', 0.7626097202301025),\n ('royal', 0.7442865371704102),\n ('crown', 0.7382649183273315),\n ('victoria', 0.728577196598053)]\n\n\n\n\nRecommending songs by embeddings\n\nimport pandas as pd\nfrom urllib import request\n\n# Get the playlist dataset file\ndata = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')\n\n# Parse the playlist dataset file. Skip the first two lines as\n# they only contain metadata\nlines = data.read().decode(\"utf-8\").split('\\n')[2:]\n\n# Remove playlists with only one song\nplaylists = [s.rstrip().split() for s in lines if len(s.split()) > 1]\n\n# Load song metadata\nsongs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')\nsongs_file = songs_file.read().decode(\"utf-8\").split('\\n')\nsongs = [s.rstrip().split('\\t') for s in songs_file]\nsongs_df = pd.DataFrame(data=songs, columns = ['id', 'title', 'artist'])\nsongs_df = songs_df.set_index('id')\n\n\nsongs_df.to_csv('chen_recs_songs.csv')\n\n\nprint( 'Playlist #1:\\n ', playlists[0], '\\n')\nprint( 'Playlist #2:\\n ', playlists[1])\n\nPlaylist #1:\n  ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '2', '42', '43', '44', '45', '46', '47', '48', '20', '49', '8', '50', '51', '52', '53', '54', '55', '56', '57', '25', '58', '59', '60', '61', '62', '3', '63', '64', '65', '66', '46', '47', '67', '2', '48', '68', '69', '70', '57', '50', '71', '72', '53', '73', '25', '74', '59', '20', '46', '75', '76', '77', '59', '20', '43'] \n\nPlaylist #2:\n  ['78', '79', '80', '3', '62', '81', '14', '82', '48', '83', '84', '17', '85', '86', '87', '88', '74', '89', '90', '91', '4', '73', '62', '92', '17', '53', '59', '93', '94', '51', '50', '27', '95', '48', '96', '97', '98', '99', '100', '57', '101', '102', '25', '103', '3', '104', '105', '106', '107', '47', '108', '109', '110', '111', '112', '113', '25', '63', '62', '114', '115', '84', '116', '117', '118', '119', '120', '121', '122', '123', '50', '70', '71', '124', '17', '85', '14', '82', '48', '125', '47', '46', '72', '53', '25', '73', '4', '126', '59', '74', '20', '43', '127', '128', '129', '13', '82', '48', '130', '131', '132', '133', '134', '135', '136', '137', '59', '46', '138', '43', '20', '139', '140', '73', '57', '70', '141', '3', '1', '74', '142', '143', '144', '145', '48', '13', '25', '146', '50', '147', '126', '59', '20', '148', '149', '150', '151', '152', '56', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '60', '176', '51', '177', '178', '179', '180', '181', '182', '183', '184', '185', '57', '186', '187', '188', '189', '190', '191', '46', '192', '193', '194', '195', '196', '197', '198', '25', '199', '200', '49', '201', '100', '202', '203', '204', '205', '206', '207', '32', '208', '209', '210']\n\n\n\nfrom gensim.models import Word2Vec\n\n# Train our Word2Vec model\nmodel = Word2Vec(\n    playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4\n)\n\n\nsong_id = 2172\n\n# Ask the model for songs similar to song #2172\nmodel.wv.most_similar(positive=str(song_id))\n\n[('3094', 0.9984354972839355),\n ('3167', 0.997356653213501),\n ('10105', 0.9970279335975647),\n ('3126', 0.9969783425331116),\n ('2976', 0.9967626333236694),\n ('3116', 0.9966928362846375),\n ('10084', 0.9965797066688538),\n ('6624', 0.9957831501960754),\n ('2704', 0.995732843875885),\n ('6658', 0.9954785108566284)]\n\n\n\nprint(songs_df.iloc[2172])\n\ntitle     Fade To Black\nartist        Metallica\nName: 2172 , dtype: object\n\n\n\nimport numpy as np\n\ndef print_recommendations(song_id):\n    similar_songs = np.array(\n        model.wv.most_similar(positive=str(song_id),topn=5)\n    )[:,0]\n    return  songs_df.iloc[similar_songs]\n\n# Extract recommendations\nprint_recommendations(2172)\n\n\n\n\n\n\n\n\ntitle\nartist\n\n\nid\n\n\n\n\n\n\n3094\nBreaking The Law\nJudas Priest\n\n\n3167\nUnchained\nVan Halen\n\n\n10105\nThree Lock Box\nSammy Hagar\n\n\n3126\nHeavy Metal\nSammy Hagar\n\n\n2976\nI Don't Know\nOzzy Osbourne\n\n\n\n\n\n\n\n\nprint_recommendations(2172)\n\n\n\n\n\n\n\n\ntitle\nartist\n\n\nid\n\n\n\n\n\n\n3094\nBreaking The Law\nJudas Priest\n\n\n3167\nUnchained\nVan Halen\n\n\n10105\nThree Lock Box\nSammy Hagar\n\n\n3126\nHeavy Metal\nSammy Hagar\n\n\n2976\nI Don't Know\nOzzy Osbourne\n\n\n\n\n\n\n\n\nprint_recommendations(842)\n\n\n\n\n\n\n\n\ntitle\nartist\n\n\nid\n\n\n\n\n\n\n27081\nGive Me Everything (w\\/ Ne-Yo, Afrojack & Nayer)\nPitbull\n\n\n413\nIf I Ruled The World (Imagine That) (w\\/ Laury...\nNas\n\n\n18844\nMurder She Wrote\nChaka Demus & Pliers\n\n\n211\nHypnotize\nThe Notorious B.I.G.\n\n\n34678\nRun De Riddim\n3 Canal\n\n\n\n\n\n\n\n\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Foundations",
      "2. Tokens and embeddings"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html",
    "href": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html",
    "title": "Chapter 4 - Text Classification",
    "section": "",
    "text": "from datasets import load_dataset\n\n# Load our data\ndata = load_dataset(\"rotten_tomatoes\")\ndata\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nDatasetDict({\n    train: Dataset({\n        features: ['text', 'label'],\n        num_rows: 8530\n    })\n    validation: Dataset({\n        features: ['text', 'label'],\n        num_rows: 1066\n    })\n    test: Dataset({\n        features: ['text', 'label'],\n        num_rows: 1066\n    })\n})\n\n\n\ndata[\"train\"][0, -1]\n\n{'text': ['the rock is destined to be the 21st century\\'s new \" conan \" and that he\\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',\n  'things really get weird , though not particularly scary : the movie is all portent and no content .'],\n 'label': [1, 0]}",
    "crumbs": [
      "Home",
      "Applications",
      "4. Text classification"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#using-a-task-specific-model",
    "href": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#using-a-task-specific-model",
    "title": "Chapter 4 - Text Classification",
    "section": "Using a Task-specific Model",
    "text": "Using a Task-specific Model\n\nfrom transformers import pipeline\n\n# Path to our HF model\nmodel_path = \"cardiffnlp/twitter-roberta-base-sentiment-latest\"\n\n# Load model into pipeline\npipe = pipeline(\n    model=model_path, tokenizer=model_path, return_all_scores=True, device=\"cuda:0\"\n)\n\n\n\n\n\n\n\nSome weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nDevice set to use cuda:0\n/home/alal/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:111: UserWarning: `return_all_scores` is now deprecated,  if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.\n  warnings.warn(\n\n\n\nimport numpy as np\nfrom tqdm import tqdm\nfrom transformers.pipelines.pt_utils import KeyDataset\n\n# Run inference\ny_pred = []\nfor output in tqdm(pipe(KeyDataset(data[\"test\"], \"text\")), total=len(data[\"test\"])):\n    negative_score = output[0][\"score\"]\n    positive_score = output[2][\"score\"]\n    assignment = np.argmax([negative_score, positive_score])\n    y_pred.append(assignment)\n\n100%|██████████| 1066/1066 [00:03<00:00, 275.29it/s]\n\n\n\nfrom sklearn.metrics import classification_report\n\n\ndef evaluate_performance(y_true, y_pred):\n    \"\"\"Create and print the classification report\"\"\"\n    performance = classification_report(\n        y_true, y_pred, target_names=[\"Negative Review\", \"Positive Review\"]\n    )\n    print(performance)\n\n\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.76      0.88      0.81       533\nPositive Review       0.86      0.72      0.78       533\n\n       accuracy                           0.80      1066\n      macro avg       0.81      0.80      0.80      1066\n   weighted avg       0.81      0.80      0.80      1066",
    "crumbs": [
      "Home",
      "Applications",
      "4. Text classification"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#classification-tasks-that-leverage-embeddings",
    "href": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#classification-tasks-that-leverage-embeddings",
    "title": "Chapter 4 - Text Classification",
    "section": "Classification Tasks that Leverage Embeddings",
    "text": "Classification Tasks that Leverage Embeddings\n\nSupervised Classification\n\nfrom sentence_transformers import SentenceTransformer\n\n# Load model\nmodel = SentenceTransformer(\"sentence-transformers/all-mpnet-base-v2\")\n\n# Convert text to embeddings\ntrain_embeddings = model.encode(data[\"train\"][\"text\"], show_progress_bar=True)\ntest_embeddings = model.encode(data[\"test\"][\"text\"], show_progress_bar=True)\n\n\n\n\n\n\n\n\ntrain_embeddings.shape\n\n(8530, 768)\n\n\n\nfrom sklearn.linear_model import LogisticRegression\n\n# Train a Logistic Regression on our train embeddings\nclf = LogisticRegression(random_state=42)\nclf.fit(train_embeddings, data[\"train\"][\"label\"])\n\nLogisticRegression(random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.LogisticRegression?Documentation for LogisticRegressioniFitted\n        \n            \n                Parameters\n                \n\n\n\n\n\n\n\n\n\npenalty penalty: {'l1', 'l2', 'elasticnet', None}, default='l2'\n\nSpecify the norm of the penalty:\n\n- `None`: no penalty is added;\n- `'l2'`: add a L2 penalty term and it is the default choice;\n- `'l1'`: add a L1 penalty term;\n- `'elasticnet'`: both L1 and L2 penalty terms are added.\n\n.. warning::\nSome penalties may not work with some solvers. See the parameter\n`solver` below, to know the compatibility between the penalty and\nsolver.\n\n.. versionadded:: 0.19\nl1 penalty with SAGA solver (allowing 'multinomial' + L1)\n\n.. deprecated:: 1.8\n`penalty` was deprecated in version 1.8 and will be removed in 1.10.\nUse `l1_ratio` instead. `l1_ratio=0` for `penalty='l2'`, `l1_ratio=1` for\n`penalty='l1'` and `l1_ratio` set to any float between 0 and 1 for\n`'penalty='elasticnet'`.\n'deprecated'\n\n\n\nC C: float, default=1.0\n\nInverse of regularization strength; must be a positive float.\nLike in support vector machines, smaller values specify stronger\nregularization. `C=np.inf` results in unpenalized logistic regression.\nFor a visual example on the effect of tuning the `C` parameter\nwith an L1 penalty, see:\n:ref:`sphx_glr_auto_examples_linear_model_plot_logistic_path.py`.\n1.0\n\n\n\nl1_ratio l1_ratio: float, default=0.0\n\nThe Elastic-Net mixing parameter, with `0 <= l1_ratio <= 1`. Setting\n`l1_ratio=1` gives a pure L1-penalty, setting `l1_ratio=0` a pure L2-penalty.\nAny value between 0 and 1 gives an Elastic-Net penalty of the form\n`l1_ratio * L1 + (1 - l1_ratio) * L2`.\n\n.. warning::\nCertain values of `l1_ratio`, i.e. some penalties, may not work with some\nsolvers. See the parameter `solver` below, to know the compatibility between\nthe penalty and solver.\n\n.. versionchanged:: 1.8\nDefault value changed from None to 0.0.\n\n.. deprecated:: 1.8\n`None` is deprecated and will be removed in version 1.10. Always use\n`l1_ratio` to specify the penalty type.\n0.0\n\n\n\ndual dual: bool, default=False\n\nDual (constrained) or primal (regularized, see also\n:ref:`this equation `) formulation. Dual formulation\nis only implemented for l2 penalty with liblinear solver. Prefer `dual=False`\nwhen n_samples > n_features.\nFalse\n\n\n\ntol tol: float, default=1e-4\n\nTolerance for stopping criteria.\n0.0001\n\n\n\nfit_intercept fit_intercept: bool, default=True\n\nSpecifies if a constant (a.k.a. bias or intercept) should be\nadded to the decision function.\nTrue\n\n\n\nintercept_scaling intercept_scaling: float, default=1\n\nUseful only when the solver `liblinear` is used\nand `self.fit_intercept` is set to `True`. In this case, `x` becomes\n`[x, self.intercept_scaling]`,\ni.e. a \"synthetic\" feature with constant value equal to\n`intercept_scaling` is appended to the instance vector.\nThe intercept becomes\n``intercept_scaling * synthetic_feature_weight``.\n\n.. note::\nThe synthetic feature weight is subject to L1 or L2\nregularization as all other features.\nTo lessen the effect of regularization on synthetic feature weight\n(and therefore on the intercept) `intercept_scaling` has to be increased.\n1\n\n\n\nclass_weight class_weight: dict or 'balanced', default=None\n\nWeights associated with classes in the form ``{class_label: weight}``.\nIf not given, all classes are supposed to have weight one.\n\nThe \"balanced\" mode uses the values of y to automatically adjust\nweights inversely proportional to class frequencies in the input data\nas ``n_samples / (n_classes * np.bincount(y))``.\n\nNote that these weights will be multiplied with sample_weight (passed\nthrough the fit method) if sample_weight is specified.\n\n.. versionadded:: 0.17\n*class_weight='balanced'*\nNone\n\n\n\nrandom_state random_state: int, RandomState instance, default=None\n\nUsed when ``solver`` == 'sag', 'saga' or 'liblinear' to shuffle the\ndata. See :term:`Glossary ` for details.\n42\n\n\n\nsolver solver: {'lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga'}, default='lbfgs'\n\nAlgorithm to use in the optimization problem. Default is 'lbfgs'.\nTo choose a solver, you might want to consider the following aspects:\n\n- 'lbfgs' is a good default solver because it works reasonably well for a wide\nclass of problems.\n- For :term:`multiclass` problems (`n_classes >= 3`), all solvers except\n'liblinear' minimize the full multinomial loss, 'liblinear' will raise an\nerror.\n- 'newton-cholesky' is a good choice for\n`n_samples` >> `n_features * n_classes`, especially with one-hot encoded\ncategorical features with rare categories. Be aware that the memory usage\nof this solver has a quadratic dependency on `n_features * n_classes`\nbecause it explicitly computes the full Hessian matrix.\n- For small datasets, 'liblinear' is a good choice, whereas 'sag'\nand 'saga' are faster for large ones;\n- 'liblinear' can only handle binary classification by default. To apply a\none-versus-rest scheme for the multiclass setting one can wrap it with the\n:class:`~sklearn.multiclass.OneVsRestClassifier`.\n\n.. warning::\nThe choice of the algorithm depends on the penalty chosen (`l1_ratio=0`\nfor L2-penalty, `l1_ratio=1` for L1-penalty and `0 < l1_ratio < 1` for\nElastic-Net) and on (multinomial) multiclass support:\n\n================= ======================== ======================\nsolver l1_ratio multinomial multiclass\n================= ======================== ======================\n'lbfgs' l1_ratio=0 yes\n'liblinear' l1_ratio=1 or l1_ratio=0 no\n'newton-cg' l1_ratio=0 yes\n'newton-cholesky' l1_ratio=0 yes\n'sag' l1_ratio=0 yes\n'saga' 0<=l1_ratio<=1 yes\n================= ======================== ======================\n\n.. note::\n'sag' and 'saga' fast convergence is only guaranteed on features\nwith approximately the same scale. You can preprocess the data with\na scaler from :mod:`sklearn.preprocessing`.\n\n.. seealso::\nRefer to the :ref:`User Guide ` for more\ninformation regarding :class:`LogisticRegression` and more specifically the\n:ref:`Table `\nsummarizing solver/penalty supports.\n\n.. versionadded:: 0.17\nStochastic Average Gradient (SAG) descent solver. Multinomial support in\nversion 0.18.\n.. versionadded:: 0.19\nSAGA solver.\n.. versionchanged:: 0.22\nThe default solver changed from 'liblinear' to 'lbfgs' in 0.22.\n.. versionadded:: 1.2\nnewton-cholesky solver. Multinomial support in version 1.6.\n'lbfgs'\n\n\n\nmax_iter max_iter: int, default=100\n\nMaximum number of iterations taken for the solvers to converge.\n100\n\n\n\nverbose verbose: int, default=0\n\nFor the liblinear and lbfgs solvers set verbose to any positive\nnumber for verbosity.\n0\n\n\n\nwarm_start warm_start: bool, default=False\n\nWhen set to True, reuse the solution of the previous call to fit as\ninitialization, otherwise, just erase the previous solution.\nUseless for liblinear solver. See :term:`the Glossary `.\n\n.. versionadded:: 0.17\n*warm_start* to support *lbfgs*, *newton-cg*, *sag*, *saga* solvers.\nFalse\n\n\n\nn_jobs n_jobs: int, default=None\n\nDoes not have any effect.\n\n.. deprecated:: 1.8\n`n_jobs` is deprecated in version 1.8 and will be removed in 1.10.\nNone\n\n\n\n\n            \n        \n    \n\n\n\n# Predict previously unseen instances\ny_pred = clf.predict(test_embeddings)\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.85      0.86      0.85       533\nPositive Review       0.86      0.85      0.85       533\n\n       accuracy                           0.85      1066\n      macro avg       0.85      0.85      0.85      1066\n   weighted avg       0.85      0.85      0.85      1066\n\n\n\nTip!\nWhat would happen if we would not use a classifier at all? Instead, we can average the embeddings per class and apply cosine similarity to predict which classes match the documents best:\n\nimport numpy as np\nimport pandas as pd\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n# Average the embeddings of all documents in each target label\ndf = pd.DataFrame(\n    np.hstack([train_embeddings, np.array(data[\"train\"][\"label\"]).reshape(-1, 1)])\n)\naveraged_target_embeddings = df.groupby(768).mean().values\n\n# Find the best matching embeddings between evaluation documents and target embeddings\nsim_matrix = cosine_similarity(test_embeddings, averaged_target_embeddings)\ny_pred = np.argmax(sim_matrix, axis=1)\n\n# Evaluate the model\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.85      0.84      0.84       533\nPositive Review       0.84      0.85      0.84       533\n\n       accuracy                           0.84      1066\n      macro avg       0.84      0.84      0.84      1066\n   weighted avg       0.84      0.84      0.84      1066\n\n\n\n\n\nZero-shot Classification\n\n# Create embeddings for our labels\nlabel_embeddings = model.encode([\"A negative review\", \"A positive review\"])\n\n\nfrom sklearn.metrics.pairwise import cosine_similarity\n\n# Find the best matching label for each document\nsim_matrix = cosine_similarity(test_embeddings, label_embeddings)\ny_pred = np.argmax(sim_matrix, axis=1)\n\n\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.78      0.77      0.78       533\nPositive Review       0.77      0.79      0.78       533\n\n       accuracy                           0.78      1066\n      macro avg       0.78      0.78      0.78      1066\n   weighted avg       0.78      0.78      0.78      1066\n\n\n\nTip!\nWhat would happen if you were to use different descriptions? Use “A very negative movie review” and “A very positive movie review” to see what happens!",
    "crumbs": [
      "Home",
      "Applications",
      "4. Text classification"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#classification-with-generative-models",
    "href": "Hands-On-Large-Language-Models/chapter04/Chapter 4 - Text Classification.html#classification-with-generative-models",
    "title": "Chapter 4 - Text Classification",
    "section": "Classification with Generative Models",
    "text": "Classification with Generative Models\n\nEncoder-decoder Models\n\n# Load our model\npipe = pipeline(\"text2text-generation\", model=\"google/flan-t5-small\", device=\"cuda:0\")\n\nDevice set to use cuda:0\n\n\n\n# Prepare our data\nprompt = \"Is the following sentence positive or negative? \"\ndata = data.map(lambda example: {\"t5\": prompt + example[\"text\"]})\ndata\n\n\n\n\n\n\n\n\n\n\nDatasetDict({\n    train: Dataset({\n        features: ['text', 'label', 't5'],\n        num_rows: 8530\n    })\n    validation: Dataset({\n        features: ['text', 'label', 't5'],\n        num_rows: 1066\n    })\n    test: Dataset({\n        features: ['text', 'label', 't5'],\n        num_rows: 1066\n    })\n})\n\n\n\n# Run inference\ny_pred = []\nfor output in tqdm(pipe(KeyDataset(data[\"test\"], \"t5\")), total=len(data[\"test\"])):\n    text = output[0][\"generated_text\"]\n    y_pred.append(0 if text == \"negative\" else 1)\n\n100%|██████████| 1066/1066 [00:19<00:00, 55.69it/s]\n\n\n\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.83      0.85      0.84       533\nPositive Review       0.85      0.83      0.84       533\n\n       accuracy                           0.84      1066\n      macro avg       0.84      0.84      0.84      1066\n   weighted avg       0.84      0.84      0.84      1066\n\n\n\n\n\nChatGPT for Classification\n\nimport openai\n\n# Create client\nclient = openai.OpenAI()\n\n\ndef chatgpt_generation(prompt, document, model=\"gpt-3.5-turbo-0125\"):\n    \"\"\"Generate an output based on a prompt and an input document.\"\"\"\n    messages = [\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": prompt.replace(\"[DOCUMENT]\", document)},\n    ]\n    chat_completion = client.chat.completions.create(\n        messages=messages, model=model, temperature=0\n    )\n    return chat_completion.choices[0].message.content\n\n\n# Define a prompt template as a base\nprompt = \"\"\"Predict whether the following document is a positive or negative movie review:\n\n[DOCUMENT]\n\nIf it is positive return 1 and if it is negative return 0. Do not give any other answers.\n\"\"\"\n\n# Predict the target using GPT\ndocument = \"unpretentious , charming , quirky , original\"\nchatgpt_generation(prompt, document)\n\n'1'\n\n\nThe next step would be to run one of OpenAI’s model against the entire evaluation dataset. However, only run this when you have sufficient tokens as this will call the API for the entire test dataset (1066 records).\n\n# You can skip this if you want to save your (free) credits\npredictions = [chatgpt_generation(prompt, doc) for doc in tqdm(data[\"test\"][\"text\"])]\n\n100%|██████████| 1066/1066 [07:44<00:00,  2.30it/s]\n\n\n\n# Extract predictions\ny_pred = [int(pred) for pred in predictions]\n\n# Evaluate performance\nevaluate_performance(data[\"test\"][\"label\"], y_pred)\n\n                 precision    recall  f1-score   support\n\nNegative Review       0.86      0.97      0.91       533\nPositive Review       0.96      0.84      0.90       533\n\n       accuracy                           0.90      1066\n      macro avg       0.91      0.90      0.90      1066\n   weighted avg       0.91      0.90      0.90      1066",
    "crumbs": [
      "Home",
      "Applications",
      "4. Text classification"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "",
    "text": "import torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"microsoft/Phi-3-mini-4k-instruct\",\n    device_map=\"cuda\",\n    torch_dtype=\"auto\",\n    trust_remote_code=False,\n)\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n# Create a pipeline\npipe = pipeline(\n    \"text-generation\",\n    model=model,\n    tokenizer=tokenizer,\n    return_full_text=False,\n    max_new_tokens=500,\n    do_sample=False,\n)\n\n`torch_dtype` is deprecated! Use `dtype` instead!\n\n\n\n\n\nDevice set to use cuda\nThe following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n\n\n\n# Prompt\nmessages = [{\"role\": \"user\", \"content\": \"Create a funny joke about chickens.\"}]\n\n# Generate the output\noutput = pipe(messages)\nprint(output[0][\"generated_text\"])\n\n Why did the chicken join the band? Because it had the drumsticks!\n\n\n\n# Apply prompt template\nprompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False)\nprint(prompt)\n\n<|user|>\nCreate a funny joke about chickens.<|end|>\n<|endoftext|>\n\n\n\n# Using a high temperature\noutput = pipe(messages, do_sample=True, temperature=1)\nprint(output[0][\"generated_text\"])\n\n Why do chickens make terrible comedians?\n\n\n\n# Using a high top_p\noutput = pipe(messages, do_sample=True, top_p=1)\nprint(output[0][\"generated_text\"])\n\n Why did the chicken join a band? Because it had the drumsticks!",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#loading-our-model",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#loading-our-model",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "",
    "text": "import torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"microsoft/Phi-3-mini-4k-instruct\",\n    device_map=\"cuda\",\n    torch_dtype=\"auto\",\n    trust_remote_code=False,\n)\ntokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n# Create a pipeline\npipe = pipeline(\n    \"text-generation\",\n    model=model,\n    tokenizer=tokenizer,\n    return_full_text=False,\n    max_new_tokens=500,\n    do_sample=False,\n)\n\n`torch_dtype` is deprecated! Use `dtype` instead!\n\n\n\n\n\nDevice set to use cuda\nThe following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n\n\n\n# Prompt\nmessages = [{\"role\": \"user\", \"content\": \"Create a funny joke about chickens.\"}]\n\n# Generate the output\noutput = pipe(messages)\nprint(output[0][\"generated_text\"])\n\n Why did the chicken join the band? Because it had the drumsticks!\n\n\n\n# Apply prompt template\nprompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False)\nprint(prompt)\n\n<|user|>\nCreate a funny joke about chickens.<|end|>\n<|endoftext|>\n\n\n\n# Using a high temperature\noutput = pipe(messages, do_sample=True, temperature=1)\nprint(output[0][\"generated_text\"])\n\n Why do chickens make terrible comedians?\n\n\n\n# Using a high top_p\noutput = pipe(messages, do_sample=True, top_p=1)\nprint(output[0][\"generated_text\"])\n\n Why did the chicken join a band? Because it had the drumsticks!",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#the-basic-ingredients-of-a-prompt",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#the-basic-ingredients-of-a-prompt",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "The Basic Ingredients of a Prompt",
    "text": "The Basic Ingredients of a Prompt",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#complex-prompt",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#complex-prompt",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Complex Prompt",
    "text": "Complex Prompt\n\n# Text to summarize which we stole from https://jalammar.github.io/illustrated-transformer/ ;)\ntext = \"\"\"In the previous post, we looked at Attention – a ubiquitous method in modern deep learning models. Attention is a concept that helped improve the performance of neural machine translation applications. In this post, we will look at The Transformer – a model that uses attention to boost the speed with which these models can be trained. The Transformer outperforms the Google Neural Machine Translation model in specific tasks. The biggest benefit, however, comes from how The Transformer lends itself to parallelization. It is in fact Google Cloud’s recommendation to use The Transformer as a reference model to use their Cloud TPU offering. So let’s try to break the model apart and look at how it functions.\nThe Transformer was proposed in the paper Attention is All You Need. A TensorFlow implementation of it is available as a part of the Tensor2Tensor package. Harvard’s NLP group created a guide annotating the paper with PyTorch implementation. In this post, we will attempt to oversimplify things a bit and introduce the concepts one by one to hopefully make it easier to understand to people without in-depth knowledge of the subject matter.\nLet’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.\nPopping open that Optimus Prime goodness, we see an encoding component, a decoding component, and connections between them.\nThe encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.\nThe encoders are all identical in structure (yet they do not share weights). Each one is broken down into two sub-layers:\nThe encoder’s inputs first flow through a self-attention layer – a layer that helps the encoder look at other words in the input sentence as it encodes a specific word. We’ll look closer at self-attention later in the post.\nThe outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position.\nThe decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar what attention does in seq2seq models).\nNow that we’ve seen the major components of the model, let’s start to look at the various vectors/tensors and how they flow between these components to turn the input of a trained model into an output.\nAs is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.\nEach word is embedded into a vector of size 512. We'll represent those vectors with these simple boxes.\nThe embedding only happens in the bottom-most encoder. The abstraction that is common to all the encoders is that they receive a list of vectors each of the size 512 – In the bottom encoder that would be the word embeddings, but in other encoders, it would be the output of the encoder that’s directly below. The size of this list is hyperparameter we can set – basically it would be the length of the longest sentence in our training dataset.\nAfter embedding the words in our input sequence, each of them flows through each of the two layers of the encoder.\nHere we begin to see one key property of the Transformer, which is that the word in each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. The feed-forward layer does not have those dependencies, however, and thus the various paths can be executed in parallel while flowing through the feed-forward layer.\nNext, we’ll switch up the example to a shorter sentence and we’ll look at what happens in each sub-layer of the encoder.\nNow We’re Encoding!\nAs we’ve mentioned already, an encoder receives a list of vectors as input. It processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network, then sends out the output upwards to the next encoder.\n\"\"\"\n\n# Prompt components\npersona = \"You are an expert in Large Language models. You excel at breaking down complex papers into digestible summaries.\\n\"\ninstruction = \"Summarize the key findings of the paper provided.\\n\"\ncontext = \"Your summary should extract the most crucial points that can help researchers quickly understand the most vital information of the paper.\\n\"\ndata_format = \"Create a bullet-point summary that outlines the method. Follow this up with a concise paragraph that encapsulates the main results.\\n\"\naudience = \"The summary is designed for busy researchers that quickly need to grasp the newest trends in Large Language Models.\\n\"\ntone = \"The tone should be professional and clear.\\n\"\ntext = \"MY TEXT TO SUMMARIZE\"  # Replace with your own text to summarize\ndata = f\"Text to summarize: {text}\"\n\n# The full prompt - remove and add pieces to view its impact on the generated output\nquery = persona + instruction + context + data_format + audience + tone + data\n\n\nmessages = [{\"role\": \"user\", \"content\": query}]\nprint(tokenizer.apply_chat_template(messages, tokenize=False))\n\n<|user|>\nYou are an expert in Large Language models. You excel at breaking down complex papers into digestible summaries.\nSummarize the key findings of the paper provided.\nYour summary should extract the most crucial points that can help researchers quickly understand the most vital information of the paper.\nCreate a bullet-point summary that outlines the method. Follow this up with a concise paragraph that encapsulates the main results.\nThe summary is designed for busy researchers that quickly need to grasp the newest trends in Large Language Models.\nThe tone should be professional and clear.\nText to summarize: MY TEXT TO SUMMARIZE<|end|>\n<|endoftext|>\n\n\n\n# Generate the output\noutputs = pipe(messages)\nprint(outputs[0][\"generated_text\"])\n\n - The paper investigates the impact of pre-training data size on the performance of Large Language Models (LLMs).\n\n- It compares models trained on different volumes of data, ranging from a few billion to several trillion tokens.\n\n- The study finds that models trained on larger datasets generally perform better on a variety of tasks, including language understanding and generation.\n\n- However, the performance gains diminish after a certain point, indicating a potential plateau in the benefits of increasing data size.\n\n- The paper also discusses the diminishing returns in terms of computational resources and environmental impact.\n\n- It suggests that future research should focus on optimizing model architecture and training procedures to achieve better performance with less data.\n\n\nThe paper presents a comprehensive analysis of how the size of pre-training data affects the efficacy of Large Language Models. It reveals that while larger datasets typically yield superior performance across multiple language tasks, there is a threshold beyond which additional data does not translate into significant improvements. This finding is crucial for the research community as it highlights the importance of optimizing model design and training strategies to balance performance gains with resource efficiency and environmental sustainability.",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#in-context-learning-providing-examples",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#in-context-learning-providing-examples",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "In-Context Learning: Providing Examples",
    "text": "In-Context Learning: Providing Examples\n\n# Use a single example of using the made-up word in a sentence\none_shot_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"A 'Gigamuru' is a type of Japanese musical instrument. An example of a sentence that uses the word Gigamuru is:\",\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": \"I have a Gigamuru that my uncle gave me as a gift. I love to play it at home.\",\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"To 'screeg' something is to swing a sword at it. An example of a sentence that uses the word screeg is:\",\n    },\n]\nprint(tokenizer.apply_chat_template(one_shot_prompt, tokenize=False))\n\n<|user|>\nA 'Gigamuru' is a type of Japanese musical instrument. An example of a sentence that uses the word Gigamuru is:<|end|>\n<|assistant|>\nI have a Gigamuru that my uncle gave me as a gift. I love to play it at home.<|end|>\n<|user|>\nTo 'screeg' something is to swing a sword at it. An example of a sentence that uses the word screeg is:<|end|>\n<|endoftext|>\n\n\n\n# Generate the output\noutputs = pipe(one_shot_prompt)\nprint(outputs[0][\"generated_text\"])\n\n During the medieval reenactment, the knight skillfully screeged the wooden target with precision and grace.",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#chain-prompting-breaking-up-the-problem",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#chain-prompting-breaking-up-the-problem",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Chain Prompting: Breaking up the Problem",
    "text": "Chain Prompting: Breaking up the Problem\n\n# Create name and slogan for a product\nproduct_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Create a name and slogan for a chatbot that leverages LLMs.\",\n    }\n]\noutputs = pipe(product_prompt)\nproduct_description = outputs[0][\"generated_text\"]\nprint(product_description)\n\n Name: ChatSage\nSlogan: \"Your AI Companion for Smart Conversations\"\n\n\n\n# Based on a name and slogan for a product, generate a sales pitch\nsales_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": f\"Generate a very short sales pitch for the following product: '{product_description}'\",\n    }\n]\noutputs = pipe(sales_prompt)\nsales_pitch = outputs[0][\"generated_text\"]\nprint(sales_pitch)\n\n Introducing ChatSage, your AI companion for smart conversations. With ChatSage, you'll have a personalized and intelligent assistant at your fingertips, ready to engage in meaningful dialogue, provide helpful information, and enhance your daily interactions. Experience the future of communication with ChatSage – your smart conversation partner.",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#chain-of-thought-think-before-answering",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#chain-of-thought-think-before-answering",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Chain-of-Thought: Think Before Answering",
    "text": "Chain-of-Thought: Think Before Answering\n\n# Answering without explicit reasoning\nstandard_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?\",\n    },\n    {\"role\": \"assistant\", \"content\": \"11\"},\n    {\n        \"role\": \"user\",\n        \"content\": \"The cafeteria had 25 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?\",\n    },\n]\n\n# Run generative model\noutputs = pipe(standard_prompt)\nprint(outputs[0][\"generated_text\"])\n\n The cafeteria started with 25 apples. They used 20 apples to make lunch, so they had 25 - 20 = 5 apples left. Then they bought 6 more apples, so they now have 5 + 6 = 11 apples.\n\n\n\n# Answering with chain-of-thought\ncot_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?\",\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": \"Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.\",\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?\",\n    },\n]\n\n# Generate the output\noutputs = pipe(cot_prompt)\nprint(outputs[0][\"generated_text\"])\n\n The cafeteria started with 23 apples. They used 20 apples for lunch, so they had 23 - 20 = 3 apples left. After buying 6 more apples, they now have 3 + 6 = 9 apples. The answer is 9.",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#zero-shot-chain-of-thought",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#zero-shot-chain-of-thought",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Zero-shot Chain-of-Thought",
    "text": "Zero-shot Chain-of-Thought\n\n# Zero-shot Chain-of-Thought\nzeroshot_cot_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? Let's think step-by-step.\",\n    }\n]\n\n# Generate the output\noutputs = pipe(zeroshot_cot_prompt)\nprint(outputs[0][\"generated_text\"])\n\n Step 1: Start with the initial number of apples in the cafeteria, which is 23.\n\nStep 2: Subtract the number of apples used to make lunch, which is 20.\n23 - 20 = 3 apples remaining.\n\nStep 3: Add the number of apples bought, which is 6.\n3 + 6 = 9 apples.\n\nSo, the cafeteria now has 9 apples.",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#tree-of-thought-exploring-intermediate-steps",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#tree-of-thought-exploring-intermediate-steps",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Tree-of-Thought: Exploring Intermediate Steps",
    "text": "Tree-of-Thought: Exploring Intermediate Steps\n\n# Zero-shot Chain-of-Thought\nzeroshot_tot_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Imagine three different experts are answering this question. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realises they're wrong at any point then they leave. The question is 'The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?' Make sure to discuss the results.\",\n    }\n]\n\n\n# Generate the output\noutputs = pipe(zeroshot_tot_prompt)\nprint(outputs[0][\"generated_text\"])\n\nYou seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n\n\n Expert 1:\nStep 1: Start with the initial number of apples, which is 23.\n\nExpert 2:\nStep 1: Subtract the number of apples used for lunch, which is 20.\nStep 2: Add the number of apples bought, which is 6.\n\nExpert 3:\nStep 1: Start with the initial number of apples, which is 23.\nStep 2: Subtract the number of apples used for lunch, which is 20.\nStep 3: Add the number of apples bought, which is 6.\n\nResults:\nAll three experts arrived at the same answer. The cafeteria has 9 apples left (23 - 20 + 6 = 9).",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#providing-examples",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#providing-examples",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Providing Examples",
    "text": "Providing Examples\n\n# Zero-shot learning: Providing no examples\nzeroshot_prompt = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Create a character profile for an RPG game in JSON format.\",\n    }\n]\n\n# Generate the output\noutputs = pipe(zeroshot_prompt)\nprint(outputs[0][\"generated_text\"])\n\n ```json\n{\n  \"name\": \"Aria Stormbringer\",\n  \"class\": \"Warrior\",\n  \"race\": \"Human\",\n  \"level\": 10,\n  \"attributes\": {\n    \"strength\": 18,\n    \"dexterity\": 12,\n    \"constitution\": 16,\n    \"intelligence\": 8,\n    \"wisdom\": 10,\n    \"charisma\": 14\n  },\n  \"skills\": {\n    \"melee\": 15,\n    \"ranged\": 10,\n    \"magic\": 5,\n    \"stealth\": 12,\n    \"acrobatics\": 10,\n    \"survival\": 14\n  },\n  \"equipment\": {\n    \"weapon\": \"Two-handed sword\",\n    \"armor\": \"Chainmail hauberk\",\n    \"shield\": \"Warhammer\",\n    \"accessories\": [\n      \"Warrior's talisman\",\n      \"Leather armor\",\n      \"Warhammer\"\n    ]\n  },\n  \"background\": \"Aria was born into a noble family, but her life took a turn when her father was killed in battle. She trained as a warrior to avenge his death and protect her homeland from invaders.\"\n}\n```\n\n\n\n# One-shot learning: Providing an example of the output structure\none_shot_template = \"\"\"Create a short character profile for an RPG game. Make sure to only use this format:\n\n{\n  \"description\": \"A SHORT DESCRIPTION\",\n  \"name\": \"THE CHARACTER'S NAME\",\n  \"armor\": \"ONE PIECE OF ARMOR\",\n  \"weapon\": \"ONE OR MORE WEAPONS\"\n}\n\"\"\"\none_shot_prompt = [{\"role\": \"user\", \"content\": one_shot_template}]\n\n# Generate the output\noutputs = pipe(one_shot_prompt)\nprint(outputs[0][\"generated_text\"])\n\n {\n  \"description\": \"A cunning rogue with a mysterious past, skilled in stealth and deception.\",\n  \"name\": \"Shadowcloak\",\n  \"armor\": \"Leather Hood\",\n  \"weapon\": \"Dagger\"\n}",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#grammar-constrained-sampling",
    "href": "Hands-On-Large-Language-Models/chapter06/Chapter 6 - Prompt Engineering.html#grammar-constrained-sampling",
    "title": "Chapter 6 - Prompt Engineering",
    "section": "Grammar: Constrained Sampling",
    "text": "Grammar: Constrained Sampling\n\nimport gc\nimport torch\n\ndel model, tokenizer, pipe\n\n# Flush memory\ngc.collect()\ntorch.cuda.empty_cache()\n\n\nfrom llama_cpp.llama import Llama\n\n# Load Phi-3\nllm = Llama.from_pretrained(\n    repo_id=\"microsoft/Phi-3-mini-4k-instruct-gguf\",\n    filename=\"*fp16.gguf\",\n    n_gpu_layers=-1,\n    n_ctx=2048,\n    verbose=False,\n)\n\n\n# Generate output\noutput = llm.create_chat_completion(\n    messages=[\n        {\"role\": \"user\", \"content\": \"Create a warrior for an RPG in JSON format.\"},\n    ],\n    response_format={\"type\": \"json_object\"},\n    temperature=0,\n)[\"choices\"][0][\"message\"][\"content\"]\n\n\nimport json\n\n# Format as json\njson_output = json.dumps(json.loads(output), indent=4)\nprint(json_output)\n\n{\n    \"name\": \"Eldrin Stormbringer\",\n    \"class\": \"Ranger\",\n    \"level\": 5,\n    \"attributes\": {\n        \"strength\": 14,\n        \"dexterity\": 18,\n        \"constitution\": 12,\n        \"intelligence\": 10,\n        \"wisdom\": 13,\n        \"charisma\": 9\n    },\n    \"skills\": {\n        \"archery\": {\n            \"proficiency\": 20,\n            \"critical_hit_chance\": 5,\n            \"damage_range\": [\n                8,\n                14\n            ]\n        },\n        \"stealth\": {\n            \"proficiency\": 17,\n            \"critical_hit_chance\": 3,\n            \"damage_range\": [\n                2,\n                6\n            ]\n        },\n        \"nature_magic\": {\n            \"proficiency\": 15,\n            \"critical_hit_chance\": 4,\n            \"healing_range\": [\n                3,\n                7\n            ],\n            \"damage_range\": [\n                -2,\n                2\n            ]\n        }\n    },\n    \"equipment\": {\n        \"weapons\": [\n            \"Longbow\",\n            \"Dagger\"\n        ],\n        \"armor\": \"Leather Armor\",\n        \"accessories\": [\n            \"Boots of Speed\",\n            \"Ring of Protection\"\n        ]\n    },\n    \"background\": \"Eldrin grew up in the wilderness, learning to hunt and track from his father. He became a skilled ranger after joining a group of adventurers.\"\n}",
    "crumbs": [
      "Home",
      "Applications",
      "6. Prompt engineering"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "import cohere\n\n# Paste your API key here. Remember to not share publicly\napi_key = ''\n\n# Create and retrieve a Cohere API key from os.cohere.ai\nco = cohere.Client(api_key)\n\n\ntext = \"\"\"\nInterstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan.\nIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine.\nSet in a dystopian future where humanity is struggling to survive, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for mankind.\n\nBrothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007.\nCaltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar.\nCinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm.\nPrincipal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles.\nInterstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects.\n\nInterstellar premiered on October 26, 2014, in Los Angeles.\nIn the United States, it was first released on film stock, expanding to venues using digital projectors.\nThe film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014.\nIt received acclaim for its performances, direction, screenplay, musical score, visual effects, ambition, themes, and emotional weight.\nIt has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics. Since its premiere, Interstellar gained a cult following,[5] and now is regarded by many sci-fi experts as one of the best science-fiction films of all time.\nInterstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades\"\"\"\n\n# Split into a list of sentences\ntexts = text.split('.')\n\n# Clean up to remove empty spaces and new lines\ntexts = [t.strip(' \\n') for t in texts]\n\n\n\n\n\nimport numpy as np\n\n# Get the embeddings\nresponse = co.embed(\n  texts=texts,\n  input_type=\"search_document\",\n).embeddings\n\nembeds = np.array(response)\nprint(embeds.shape)\n\n(15, 4096)\n\n\n\n\n\n\nimport faiss\n\ndim = embeds.shape[1]\nindex = faiss.IndexFlatL2(dim)\nindex.add(np.float32(embeds))\n\n\n\n\n\nimport pandas as pd\n\ndef search(query, number_of_results=3):\n\n  # 1. Get the query's embedding\n  query_embed = co.embed(texts=[query],\n                input_type=\"search_query\",).embeddings[0]\n\n  # 2. Retrieve the nearest neighbors\n  distances , similar_item_ids = index.search(np.float32([query_embed]), number_of_results)\n\n  # 3. Format the results\n  texts_np = np.array(texts) # Convert texts list to numpy for easier indexing\n  results = pd.DataFrame(data={'texts': texts_np[similar_item_ids[0]],\n                              'distance': distances[0]})\n\n  # 4. Print and return the results\n  print(f\"Query:'{query}'\\nNearest neighbors:\")\n  return results\n\n\nquery = \"how precise was the science\"\nresults = search(query)\nresults\n\nQuery:'how precise was the science'\nNearest neighbors:\n\n\n                                               texts      distance\n0  It has also received praise from many astronom...  10757.379883\n1  Caltech theoretical physicist and 2017 Nobel l...  11566.131836\n2  Interstellar uses extensive practical and mini...  11922.833008\n\n\n\nfrom rank_bm25 import BM25Okapi\nfrom sklearn.feature_extraction import _stop_words\nimport string\n\ndef bm25_tokenizer(text):\n    tokenized_doc = []\n    for token in text.lower().split():\n        token = token.strip(string.punctuation)\n\n        if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:\n            tokenized_doc.append(token)\n    return tokenized_doc\n\n\nfrom tqdm import tqdm\n\ntokenized_corpus = []\nfor passage in tqdm(texts):\n    tokenized_corpus.append(bm25_tokenizer(passage))\n\nbm25 = BM25Okapi(tokenized_corpus)\n\n100%|██████████| 15/15 [00:00<00:00, 38908.20it/s]\n\n\n\ndef keyword_search(query, top_k=3, num_candidates=15):\n    print(\"Input question:\", query)\n\n    ##### BM25 search (lexical search) #####\n    bm25_scores = bm25.get_scores(bm25_tokenizer(query))\n    top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]\n    bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]\n    bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)\n\n    print(f\"Top-3 lexical search (BM25) hits\")\n    for hit in bm25_hits[0:top_k]:\n        print(\"\\t{:.3f}\\t{}\".format(hit['score'], texts[hit['corpus_id']].replace(\"\\n\", \" \")))\n\n\nkeyword_search(query = \"how precise was the science\")\n\nInput question: how precise was the science\nTop-3 lexical search (BM25) hits\n    1.789   Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan\n    1.373   Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n    0.000   It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine\n\n\n\n\n\n\nquery = \"What is the mass of the moon?\"\nresults = search(query)\nresults\n\nQuery:'What is the mass of the moon?'\nNearest neighbors:\n\n\n                                               texts      distance\n0  Cinematographer Hoyte van Hoytema shot it on 3...  12854.458984\n1  The film had a worldwide gross over $677 milli...  13301.030273\n2  It has also received praise from many astronom...  13332.011719",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#getting-the-text-archive-and-chunking-it",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#getting-the-text-archive-and-chunking-it",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "import cohere\n\n# Paste your API key here. Remember to not share publicly\napi_key = ''\n\n# Create and retrieve a Cohere API key from os.cohere.ai\nco = cohere.Client(api_key)\n\n\ntext = \"\"\"\nInterstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan.\nIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine.\nSet in a dystopian future where humanity is struggling to survive, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for mankind.\n\nBrothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007.\nCaltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar.\nCinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm.\nPrincipal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles.\nInterstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects.\n\nInterstellar premiered on October 26, 2014, in Los Angeles.\nIn the United States, it was first released on film stock, expanding to venues using digital projectors.\nThe film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014.\nIt received acclaim for its performances, direction, screenplay, musical score, visual effects, ambition, themes, and emotional weight.\nIt has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics. Since its premiere, Interstellar gained a cult following,[5] and now is regarded by many sci-fi experts as one of the best science-fiction films of all time.\nInterstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades\"\"\"\n\n# Split into a list of sentences\ntexts = text.split('.')\n\n# Clean up to remove empty spaces and new lines\ntexts = [t.strip(' \\n') for t in texts]",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#embedding-the-text-chunks",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#embedding-the-text-chunks",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "import numpy as np\n\n# Get the embeddings\nresponse = co.embed(\n  texts=texts,\n  input_type=\"search_document\",\n).embeddings\n\nembeds = np.array(response)\nprint(embeds.shape)\n\n(15, 4096)",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#building-the-search-index",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#building-the-search-index",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "import faiss\n\ndim = embeds.shape[1]\nindex = faiss.IndexFlatL2(dim)\nindex.add(np.float32(embeds))",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#search-the-index",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#search-the-index",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "import pandas as pd\n\ndef search(query, number_of_results=3):\n\n  # 1. Get the query's embedding\n  query_embed = co.embed(texts=[query],\n                input_type=\"search_query\",).embeddings[0]\n\n  # 2. Retrieve the nearest neighbors\n  distances , similar_item_ids = index.search(np.float32([query_embed]), number_of_results)\n\n  # 3. Format the results\n  texts_np = np.array(texts) # Convert texts list to numpy for easier indexing\n  results = pd.DataFrame(data={'texts': texts_np[similar_item_ids[0]],\n                              'distance': distances[0]})\n\n  # 4. Print and return the results\n  print(f\"Query:'{query}'\\nNearest neighbors:\")\n  return results\n\n\nquery = \"how precise was the science\"\nresults = search(query)\nresults\n\nQuery:'how precise was the science'\nNearest neighbors:\n\n\n                                               texts      distance\n0  It has also received praise from many astronom...  10757.379883\n1  Caltech theoretical physicist and 2017 Nobel l...  11566.131836\n2  Interstellar uses extensive practical and mini...  11922.833008\n\n\n\nfrom rank_bm25 import BM25Okapi\nfrom sklearn.feature_extraction import _stop_words\nimport string\n\ndef bm25_tokenizer(text):\n    tokenized_doc = []\n    for token in text.lower().split():\n        token = token.strip(string.punctuation)\n\n        if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:\n            tokenized_doc.append(token)\n    return tokenized_doc\n\n\nfrom tqdm import tqdm\n\ntokenized_corpus = []\nfor passage in tqdm(texts):\n    tokenized_corpus.append(bm25_tokenizer(passage))\n\nbm25 = BM25Okapi(tokenized_corpus)\n\n100%|██████████| 15/15 [00:00<00:00, 38908.20it/s]\n\n\n\ndef keyword_search(query, top_k=3, num_candidates=15):\n    print(\"Input question:\", query)\n\n    ##### BM25 search (lexical search) #####\n    bm25_scores = bm25.get_scores(bm25_tokenizer(query))\n    top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]\n    bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]\n    bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)\n\n    print(f\"Top-3 lexical search (BM25) hits\")\n    for hit in bm25_hits[0:top_k]:\n        print(\"\\t{:.3f}\\t{}\".format(hit['score'], texts[hit['corpus_id']].replace(\"\\n\", \" \")))\n\n\nkeyword_search(query = \"how precise was the science\")\n\nInput question: how precise was the science\nTop-3 lexical search (BM25) hits\n    1.789   Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan\n    1.373   Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n    0.000   It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#caveats-of-dense-retrieval",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#caveats-of-dense-retrieval",
    "title": "Chapter 8 - Semantic Search",
    "section": "",
    "text": "query = \"What is the mass of the moon?\"\nresults = search(query)\nresults\n\nQuery:'What is the mass of the moon?'\nNearest neighbors:\n\n\n                                               texts      distance\n0  Cinematographer Hoyte van Hoytema shot it on 3...  12854.458984\n1  The film had a worldwide gross over $677 milli...  13301.030273\n2  It has also received praise from many astronom...  13332.011719",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#example-grounded-generation-with-an-llm-api",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#example-grounded-generation-with-an-llm-api",
    "title": "Chapter 8 - Semantic Search",
    "section": "Example: Grounded Generation with an LLM API",
    "text": "Example: Grounded Generation with an LLM API\n\nquery = \"income generated\"\n\n# 1- Retrieval\n# We'll use embedding search. But ideally we'd do hybrid\nresults = search(query)\n\n# 2- Grounded Generation\ndocs_dict = [{'text': text} for text in results['texts']]\nresponse = co.chat(\n    message = query,\n    documents=docs_dict\n)\n\nprint(response.text)\n\nQuery:'income generated'\nNearest neighbors:\nThe film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.\n\n\n\nresponse\n\nNonStreamedChatResponse(text='The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.', generation_id='bebc32bd-d620-42cf-bd13-f8d1f96d4aa6', citations=[ChatCitation(start=21, end=57, text='worldwide gross of over $677 million', document_ids=['doc_0']), ChatCitation(start=62, end=103, text='$773 million with subsequent re-releases.', document_ids=['doc_0'])], documents=[{'id': 'doc_0', 'text': 'The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014'}], is_search_required=None, search_queries=None, search_results=None, finish_reason='COMPLETE', tool_calls=None, chat_history=[Message_User(message='income generated', tool_calls=None, role='USER'), Message_Chatbot(message='The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.', tool_calls=None, role='CHATBOT')], prompt=None, meta=ApiMeta(api_version=ApiMetaApiVersion(version='1', is_deprecated=None, is_experimental=None), billed_units=ApiMetaBilledUnits(input_tokens=106, output_tokens=26, search_units=None, classifications=None), tokens=ApiMetaTokens(input_tokens=797, output_tokens=95), warnings=None), response_id='319002db-505d-4d3c-903f-41eefbf0f856')\n\n\n\nresponse.citations\n\n[ChatCitation(start=21, end=57, text='worldwide gross of over $677 million', document_ids=['doc_0']),\n ChatCitation(start=62, end=103, text='$773 million with subsequent re-releases.', document_ids=['doc_0'])]",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#example-rag-with-local-models",
    "href": "Hands-On-Large-Language-Models/chapter08/Chapter 8 - Semantic Search.html#example-rag-with-local-models",
    "title": "Chapter 8 - Semantic Search",
    "section": "Example: RAG with Local Models",
    "text": "Example: RAG with Local Models\n\nLoading the Generation Model\n\n!wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf\n\n--2024-06-21 09:49:22--  https://huggingface.co/lmstudio-community/Phi-3-mini-4k-instruct-GGUF/resolve/main/Phi-3-mini-4k-instruct-Q8_0.gguf\nResolving huggingface.co (huggingface.co)... 18.164.174.118, 18.164.174.23, 18.164.174.17, ...\nConnecting to huggingface.co (huggingface.co)|18.164.174.118|:443... connected.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://cdn-lfs-us-1.huggingface.co/repos/8e/3f/8e3fafa0351929e621a3db9a53b131a9d7f4b222332208032555bb92f11ab100/8d2f3732e31c354e169cd81dcde9807a1c73b85b9a0f9b16c19013e7a4bb151c?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-Q8_0.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-Q8_0.gguf%22%3B&Expires=1719222562&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxOTIyMjU2Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhlLzNmLzhlM2ZhZmEwMzUxOTI5ZTYyMWEzZGI5YTUzYjEzMWE5ZDdmNGIyMjIzMzIyMDgwMzI1NTViYjkyZjExYWIxMDAvOGQyZjM3MzJlMzFjMzU0ZTE2OWNkODFkY2RlOTgwN2ExYzczYjg1YjlhMGY5YjE2YzE5MDEzZTdhNGJiMTUxYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=DFqGVAGZBHhpPlWt2pSH5nbTgskxr6FrSK19FhopbgIYnqujHL3LDgnleMTDHrtRvflyX-6QE88ZAYZc1wIZA7ZXgBVoFdYMNPbHixfv4hqGdKiddWcF4QYi5JCYChS3Z8oZPUkcbyX6KNTqMR1nls2KTZ3K0Xl3E7nGmlTXo85mRdRKQojZLkuLOa28pG2z9jrs1wJ1B2W3Ed%7E%7EK1E-BXhjKsK4zUR1Ch-3KfqAqe0q0XmnCcF1Ml2xosujvlA%7EZuVxflmRf8wRVZBZsbZNXaUFAb3oiyNTuyr9g1fvSxJPmAJocs9jIMUZwU88Sa4s%7EVdfytD0s6YruNH1GOAfoA__&Key-Pair-Id=K2FPYV99P2N66Q [following]\n--2024-06-21 09:49:22--  https://cdn-lfs-us-1.huggingface.co/repos/8e/3f/8e3fafa0351929e621a3db9a53b131a9d7f4b222332208032555bb92f11ab100/8d2f3732e31c354e169cd81dcde9807a1c73b85b9a0f9b16c19013e7a4bb151c?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-Q8_0.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-Q8_0.gguf%22%3B&Expires=1719222562&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxOTIyMjU2Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhlLzNmLzhlM2ZhZmEwMzUxOTI5ZTYyMWEzZGI5YTUzYjEzMWE5ZDdmNGIyMjIzMzIyMDgwMzI1NTViYjkyZjExYWIxMDAvOGQyZjM3MzJlMzFjMzU0ZTE2OWNkODFkY2RlOTgwN2ExYzczYjg1YjlhMGY5YjE2YzE5MDEzZTdhNGJiMTUxYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=DFqGVAGZBHhpPlWt2pSH5nbTgskxr6FrSK19FhopbgIYnqujHL3LDgnleMTDHrtRvflyX-6QE88ZAYZc1wIZA7ZXgBVoFdYMNPbHixfv4hqGdKiddWcF4QYi5JCYChS3Z8oZPUkcbyX6KNTqMR1nls2KTZ3K0Xl3E7nGmlTXo85mRdRKQojZLkuLOa28pG2z9jrs1wJ1B2W3Ed%7E%7EK1E-BXhjKsK4zUR1Ch-3KfqAqe0q0XmnCcF1Ml2xosujvlA%7EZuVxflmRf8wRVZBZsbZNXaUFAb3oiyNTuyr9g1fvSxJPmAJocs9jIMUZwU88Sa4s%7EVdfytD0s6YruNH1GOAfoA__&Key-Pair-Id=K2FPYV99P2N66Q\nResolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 18.65.25.71, 18.65.25.64, 18.65.25.113, ...\nConnecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|18.65.25.71|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 4061221024 (3.8G) [binary/octet-stream]\nSaving to: ‘Phi-3-mini-4k-instruct-Q8_0.gguf’\n\nPhi-3-mini-4k-instr 100%[===================>]   3.78G  66.7MB/s    in 33s     \n\n2024-06-21 09:49:55 (119 MB/s) - ‘Phi-3-mini-4k-instruct-Q8_0.gguf’ saved [4061221024/4061221024]\n\n\n\n\nfrom langchain import LlamaCpp\n\n# Make sure the model path is correct for your system!\nllm = LlamaCpp(\n    model_path=\"Phi-3-mini-4k-instruct-q4.gguf\",\n    n_gpu_layers=-1,\n    max_tokens=500,\n    n_ctx=2048,\n    seed=42,\n    verbose=False\n)\n\n\n\nLoading the Embedding Model\n\nfrom langchain.embeddings.huggingface import HuggingFaceEmbeddings\n\n# Embedding Model for converting text to numerical representations\nembedding_model = HuggingFaceEmbeddings(\n    model_name='BAAI/bge-small-en-v1.5'\n)\n\n\n\n\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPreparing the Vector Database\n\nfrom langchain.vectorstores import FAISS\n\n# Create a local vector database\ndb = FAISS.from_texts(texts, embedding_model)\n\n\n\nThe RAG Prompt\n\nfrom langchain import PromptTemplate\nfrom langchain.chains import RetrievalQA\n\n\n# Create a prompt template\ntemplate = \"\"\"<|user|>\nRelevant information:\n{context}\n\nProvide a concise answer the following question using the relevant information provided above:\n{question}<|end|>\n<|assistant|>\"\"\"\nprompt = PromptTemplate(\n    template=template,\n    input_variables=[\"context\", \"question\"]\n)\n\n# RAG Pipeline\nrag = RetrievalQA.from_chain_type(\n    llm=llm,\n    chain_type='stuff',\n    retriever=db.as_retriever(),\n    chain_type_kwargs={\n        \"prompt\": prompt\n    },\n    verbose=True\n)\n\n\nrag.invoke('Income generated')\n\n\n\n> Entering new RetrievalQA chain...\n\n> Finished chain.\n\n\n{'query': 'Income generated',\n 'result': \" Interstellar grossed over $677 million worldwide in 2014 and had additional earnings from subsequent re-releases, totaling approximately $773 million. The film's release utilized both traditional film stock and digital projectors across various venues to maximize its income generation potential.\"}",
    "crumbs": [
      "Home",
      "Applications",
      "8. Semantic search"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from datasets import load_dataset\n\n# Load MNLI dataset from GLUE\n# 0 = entailment, 1 = neutral, 2 = contradiction\ntrain_dataset = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(50_000))\ntrain_dataset = train_dataset.remove_columns(\"idx\")\n\n\ntrain_dataset[2]\n\n{'premise': 'One of our number will carry out your instructions minutely.',\n 'hypothesis': 'A member of my team will execute your orders with immense precision.',\n 'label': 0}\n\n\n\n\n\n\nfrom sentence_transformers import SentenceTransformer\n\n# Use a base model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\nWARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name bert-base-uncased. Creating a new one with mean pooling.\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\n\nfrom sentence_transformers import losses\n\n# Define the loss function. In soft-max loss, we will also need to explicitly set the number of labels.\ntrain_loss = losses.SoftmaxLoss(\n    model=embedding_model,\n    sentence_embedding_dimension=embedding_model.get_sentence_embedding_dimension(),\n    num_labels=3\n)\n\n\n\n\n\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\",\n)\n\n\n\n\n\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"base_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\n\n# Train embedding model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n    \n      \n      \n      [1563/1563 06:10, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n100\n1.080700\n\n\n200\n0.959400\n\n\n300\n0.916200\n\n\n400\n0.870200\n\n\n500\n0.849100\n\n\n600\n0.854200\n\n\n700\n0.835200\n\n\n800\n0.825200\n\n\n900\n0.818100\n\n\n1000\n0.800300\n\n\n1100\n0.781600\n\n\n1200\n0.777100\n\n\n1300\n0.786600\n\n\n1400\n0.767900\n\n\n1500\n0.797100\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTrainOutput(global_step=1563, training_loss=0.8453957184872716, metrics={'train_runtime': 372.5713, 'train_samples_per_second': 134.202, 'train_steps_per_second': 4.195, 'total_flos': 0.0, 'train_loss': 0.8453957184872716, 'epoch': 1.0})\n\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n{'pearson_cosine': 0.3710938716460552,\n 'spearman_cosine': 0.45148122260403883,\n 'pearson_manhattan': 0.4037396904694362,\n 'spearman_manhattan': 0.4396893995197567,\n 'pearson_euclidean': 0.390788259199341,\n 'spearman_euclidean': 0.43444104358464286,\n 'pearson_dot': 0.3392927926047231,\n 'spearman_dot': 0.3530708415227247,\n 'pearson_max': 0.4037396904694362,\n 'spearman_max': 0.45148122260403883}",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#data",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#data",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from datasets import load_dataset\n\n# Load MNLI dataset from GLUE\n# 0 = entailment, 1 = neutral, 2 = contradiction\ntrain_dataset = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(50_000))\ntrain_dataset = train_dataset.remove_columns(\"idx\")\n\n\ntrain_dataset[2]\n\n{'premise': 'One of our number will carry out your instructions minutely.',\n 'hypothesis': 'A member of my team will execute your orders with immense precision.',\n 'label': 0}",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#model",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#model",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from sentence_transformers import SentenceTransformer\n\n# Use a base model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\nWARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name bert-base-uncased. Creating a new one with mean pooling.\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#loss-function",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#loss-function",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from sentence_transformers import losses\n\n# Define the loss function. In soft-max loss, we will also need to explicitly set the number of labels.\ntrain_loss = losses.SoftmaxLoss(\n    model=embedding_model,\n    sentence_embedding_dimension=embedding_model.get_sentence_embedding_dimension(),\n    num_labels=3\n)",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#evaluation",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#evaluation",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\",\n)",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#training",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#training",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "",
    "text": "from sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"base_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\n\n# Train embedding model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n    \n      \n      \n      [1563/1563 06:10, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n100\n1.080700\n\n\n200\n0.959400\n\n\n300\n0.916200\n\n\n400\n0.870200\n\n\n500\n0.849100\n\n\n600\n0.854200\n\n\n700\n0.835200\n\n\n800\n0.825200\n\n\n900\n0.818100\n\n\n1000\n0.800300\n\n\n1100\n0.781600\n\n\n1200\n0.777100\n\n\n1300\n0.786600\n\n\n1400\n0.767900\n\n\n1500\n0.797100\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTrainOutput(global_step=1563, training_loss=0.8453957184872716, metrics={'train_runtime': 372.5713, 'train_samples_per_second': 134.202, 'train_steps_per_second': 4.195, 'total_flos': 0.0, 'train_loss': 0.8453957184872716, 'epoch': 1.0})\n\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n{'pearson_cosine': 0.3710938716460552,\n 'spearman_cosine': 0.45148122260403883,\n 'pearson_manhattan': 0.4037396904694362,\n 'spearman_manhattan': 0.4396893995197567,\n 'pearson_euclidean': 0.390788259199341,\n 'spearman_euclidean': 0.43444104358464286,\n 'pearson_dot': 0.3392927926047231,\n 'spearman_dot': 0.3530708415227247,\n 'pearson_max': 0.4037396904694362,\n 'spearman_max': 0.45148122260403883}",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#cosine-similarity-loss",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#cosine-similarity-loss",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "Cosine Similarity Loss",
    "text": "Cosine Similarity Loss\n\nfrom datasets import Dataset, load_dataset\n\n# Load MNLI dataset from GLUE\n# 0 = entailment, 1 = neutral, 2 = contradiction\ntrain_dataset = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(50_000))\ntrain_dataset = train_dataset.remove_columns(\"idx\")\n\n# (neutral/contradiction)=0 and (entailment)=1\nmapping = {2: 0, 1: 0, 0:1}\ntrain_dataset = Dataset.from_dict({\n    \"sentence1\": train_dataset[\"premise\"],\n    \"sentence2\": train_dataset[\"hypothesis\"],\n    \"label\": [float(mapping[label]) for label in train_dataset[\"label\"]]\n})\n\n\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\"\n)\n\n\nfrom sentence_transformers import losses, SentenceTransformer\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\n# Loss function\ntrain_loss = losses.CosineSimilarityLoss(model=embedding_model)\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"cosineloss_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\nWARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name bert-base-uncased. Creating a new one with mean pooling.\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n    \n      \n      \n      [1563/1563 06:04, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n100\n0.231900\n\n\n200\n0.168900\n\n\n300\n0.170900\n\n\n400\n0.157800\n\n\n500\n0.152900\n\n\n600\n0.156100\n\n\n700\n0.149300\n\n\n800\n0.154500\n\n\n900\n0.150900\n\n\n1000\n0.145600\n\n\n1100\n0.147800\n\n\n1200\n0.145600\n\n\n1300\n0.145100\n\n\n1400\n0.142000\n\n\n1500\n0.141600\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTrainOutput(global_step=1563, training_loss=0.15676780793427353, metrics={'train_runtime': 364.4779, 'train_samples_per_second': 137.183, 'train_steps_per_second': 4.288, 'total_flos': 0.0, 'train_loss': 0.15676780793427353, 'epoch': 1.0})\n\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n{'pearson_cosine': 0.7222320710908391,\n 'spearman_cosine': 0.725059765496038,\n 'pearson_manhattan': 0.7338172618636865,\n 'spearman_manhattan': 0.7323465534428775,\n 'pearson_euclidean': 0.7332726423686017,\n 'spearman_euclidean': 0.7316943270141215,\n 'pearson_dot': 0.6603672299249149,\n 'spearman_dot': 0.6624301208511642,\n 'pearson_max': 0.7338172618636865,\n 'spearman_max': 0.7323465534428775}\n\n\n⚠️ VRAM Clean-up * Restart the notebook in order to clean-up memory if you move on to the next training example.\n\nimport gc\nimport torch\n\ngc.collect()\ntorch.cuda.empty_cache()",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#multiple-negatives-ranking-loss",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#multiple-negatives-ranking-loss",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "Multiple Negatives Ranking Loss",
    "text": "Multiple Negatives Ranking Loss\n\nimport random\nfrom tqdm import tqdm\nfrom datasets import Dataset, load_dataset\n\n# # Load MNLI dataset from GLUE\nmnli = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(50_000))\nmnli = mnli.remove_columns(\"idx\")\nmnli = mnli.filter(lambda x: True if x['label'] == 0 else False)\n\n# Prepare data and add a soft negative\ntrain_dataset = {\"anchor\": [], \"positive\": [], \"negative\": []}\nsoft_negatives = mnli[\"hypothesis\"]\nrandom.shuffle(soft_negatives)\nfor row, soft_negative in tqdm(zip(mnli, soft_negatives)):\n    train_dataset[\"anchor\"].append(row[\"premise\"])\n    train_dataset[\"positive\"].append(row[\"hypothesis\"])\n    train_dataset[\"negative\"].append(soft_negative)\ntrain_dataset = Dataset.from_dict(train_dataset)\nlen(train_dataset)\n\n16875it [00:01, 14110.96it/s]\n\n\n16875\n\n\n\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\"\n)\n\n\nfrom sentence_transformers import losses, SentenceTransformer\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\n# Loss function\ntrain_loss = losses.MultipleNegativesRankingLoss(model=embedding_model)\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"mnrloss_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\nWARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name bert-base-uncased. Creating a new one with mean pooling.\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n    \n      \n      \n      [528/528 03:00, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n100\n0.345200\n\n\n200\n0.105500\n\n\n300\n0.079000\n\n\n400\n0.062200\n\n\n500\n0.069000\n\n\n\n\n\n\n\n\n\nTrainOutput(global_step=528, training_loss=0.12795479415041028, metrics={'train_runtime': 180.866, 'train_samples_per_second': 93.301, 'train_steps_per_second': 2.919, 'total_flos': 0.0, 'train_loss': 0.12795479415041028, 'epoch': 1.0})\n\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n{'pearson_cosine': 0.8070727434643791,\n 'spearman_cosine': 0.8106193672462586,\n 'pearson_manhattan': 0.8213132116968124,\n 'spearman_manhattan': 0.8164551132664518,\n 'pearson_euclidean': 0.820988086354926,\n 'spearman_euclidean': 0.8160139830687847,\n 'pearson_dot': 0.7429357515240518,\n 'spearman_dot': 0.7316164586329814,\n 'pearson_max': 0.8213132116968124,\n 'spearman_max': 0.8164551132664518}",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#supervised",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#supervised",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "Supervised",
    "text": "Supervised\n\nfrom datasets import load_dataset\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Load MNLI dataset from GLUE\n# 0 = entailment, 1 = neutral, 2 = contradiction\ntrain_dataset = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(50_000))\ntrain_dataset = train_dataset.remove_columns(\"idx\")\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\"\n)\n\n\nfrom sentence_transformers import losses, SentenceTransformer\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define model\nembedding_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\n\n# Loss function\ntrain_loss = losses.MultipleNegativesRankingLoss(model=embedding_model)\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"finetuned_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n\n\n\n\n\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n    \n      \n      \n      [1563/1563 01:57, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n100\n0.155500\n\n\n200\n0.110000\n\n\n300\n0.118600\n\n\n400\n0.115300\n\n\n500\n0.110700\n\n\n600\n0.101000\n\n\n700\n0.113100\n\n\n800\n0.099800\n\n\n900\n0.109600\n\n\n1000\n0.105800\n\n\n1100\n0.094900\n\n\n1200\n0.106400\n\n\n1300\n0.105300\n\n\n1400\n0.105200\n\n\n1500\n0.106600\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTrainOutput(global_step=1563, training_loss=0.10982195932897176, metrics={'train_runtime': 117.3739, 'train_samples_per_second': 425.989, 'train_steps_per_second': 13.316, 'total_flos': 0.0, 'train_loss': 0.10982195932897176, 'epoch': 1.0})\n\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n{'pearson_cosine': 0.8489503881223601,\n 'spearman_cosine': 0.8484667083117318,\n 'pearson_manhattan': 0.8503843871673679,\n 'spearman_manhattan': 0.8475679105384369,\n 'pearson_euclidean': 0.8513072191805562,\n 'spearman_euclidean': 0.8484667083117318,\n 'pearson_dot': 0.8489503890256918,\n 'spearman_dot': 0.8484667083117318,\n 'pearson_max': 0.8513072191805562,\n 'spearman_max': 0.8484667083117318}\n\n\n\n# Evaluate the pre-trained model\noriginal_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\nevaluator(original_model)\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n{'pearson_cosine': 0.8696194608752055,\n 'spearman_cosine': 0.8671637433378804,\n 'pearson_manhattan': 0.8670399009851635,\n 'spearman_manhattan': 0.8663946139224048,\n 'pearson_euclidean': 0.867871599362501,\n 'spearman_euclidean': 0.8671643653432983,\n 'pearson_dot': 0.8696194616795601,\n 'spearman_dot': 0.8671631197908374,\n 'pearson_max': 0.8696194616795601,\n 'spearman_max': 0.8671643653432983}\n\n\n⚠️ VRAM Clean-up * Restart the notebook in order to clean-up memory if you move on to the next training example.\n\nimport gc\nimport torch\n\ngc.collect()\ntorch.cuda.empty_cache()",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#augmented-sbert",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#augmented-sbert",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "Augmented SBERT",
    "text": "Augmented SBERT\nStep 1: Fine-tune a cross-encoder\n\nimport pandas as pd\nfrom tqdm import tqdm\nfrom datasets import load_dataset, Dataset\nfrom sentence_transformers import InputExample\nfrom sentence_transformers.datasets import NoDuplicatesDataLoader\n\n# Prepare a small set of 10000 documents for the cross-encoder\ndataset = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(10_000))\nmapping = {2: 0, 1: 0, 0:1}\n\n# Data Loader\ngold_examples = [\n    InputExample(texts=[row[\"premise\"], row[\"hypothesis\"]], label=mapping[row[\"label\"]])\n    for row in tqdm(dataset)\n]\ngold_dataloader = NoDuplicatesDataLoader(gold_examples, batch_size=32)\n\n# Pandas DataFrame for easier data handling\ngold = pd.DataFrame(\n    {\n    'sentence1': dataset['premise'],\n    'sentence2': dataset['hypothesis'],\n    'label': [mapping[label] for label in dataset['label']]\n    }\n)\n\n100%|██████████| 10000/10000 [00:00<00:00, 25870.92it/s]\n\n\n\nfrom sentence_transformers.cross_encoder import CrossEncoder\n\n# Train a cross-encoder on the gold dataset\ncross_encoder = CrossEncoder('bert-base-uncased', num_labels=2)\ncross_encoder.fit(\n    train_dataloader=gold_dataloader,\n    epochs=1,\n    show_progress_bar=True,\n    warmup_steps=100,\n    use_amp=False\n)\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\nSome weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n\n\n\n\n\n\n\n\nStep 2: Create new sentence pairs\n\n# Prepare the silver dataset by predicting labels with the cross-encoder\nsilver = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(10_000, 50_000))\npairs = list(zip(silver['premise'], silver['hypothesis']))\n\nStep 3: Label new sentence pairs with the fine-tuned cross-encoder (silver dataset)\n\nimport numpy as np\n\n# Label the sentence pairs using our fine-tuned cross-encoder\noutput = cross_encoder.predict(pairs, apply_softmax=True, show_progress_bar=True)\nsilver = pd.DataFrame(\n    {\n        \"sentence1\": silver[\"premise\"],\n        \"sentence2\": silver[\"hypothesis\"],\n        \"label\": np.argmax(output, axis=1)\n    }\n)\n\nStep 4: Train a bi-encoder (SBERT) on the extended dataset (gold + silver dataset)\n\n# Combine gold + silver\ndata = pd.concat([gold, silver], ignore_index=True, axis=0)\ndata = data.drop_duplicates(subset=['sentence1', 'sentence2'], keep=\"first\")\ntrain_dataset = Dataset.from_pandas(data, preserve_index=False)\n\n\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\"\n)\n\n\nfrom sentence_transformers import losses, SentenceTransformer\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\n# Loss function\ntrain_loss = losses.CosineSimilarityLoss(model=embedding_model)\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"augmented_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n\ntrainer.accelerator.clear()\n\nStep 5: Evaluate without silver dataset\n\n# Combine gold + silver\ndata = pd.concat([gold], ignore_index=True, axis=0)\ndata = data.drop_duplicates(subset=['sentence1', 'sentence2'], keep=\"first\")\ntrain_dataset = Dataset.from_pandas(data, preserve_index=False)\n\n# Define model\nembedding_model = SentenceTransformer('bert-base-uncased')\n\n# Loss function\ntrain_loss = losses.CosineSimilarityLoss(model=embedding_model)\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"gold_only_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\nCompared to using both the silver and gold datasets, using only the gold dataset reduces the performance of the model!\n⚠️ VRAM Clean-up * Restart the notebook in order to clean-up memory if you move on to the next training example.\n\nimport gc\nimport torch\n\ngc.collect()\ntorch.cuda.empty_cache()",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#unsupervised-learning",
    "href": "Hands-On-Large-Language-Models/chapter10/Chapter 10 - Creating Text Embedding Models.html#unsupervised-learning",
    "title": "Chapter 10 - Creating Text Embedding Models",
    "section": "Unsupervised Learning",
    "text": "Unsupervised Learning\n\nTranformer-based Denoising AutoEncoder (TSDAE)\n\n# Download additional tokenizer\nimport nltk\nnltk.download('punkt')\n\n\nfrom tqdm import tqdm\nfrom datasets import Dataset, load_dataset\nfrom sentence_transformers.datasets import DenoisingAutoEncoderDataset\n\n# Create a flat list of sentences\nmnli = load_dataset(\"glue\", \"mnli\", split=\"train\").select(range(25_000))\nflat_sentences = mnli[\"premise\"] + mnli[\"hypothesis\"]\n\n# Add noise to our input data\ndamaged_data = DenoisingAutoEncoderDataset(list(set(flat_sentences)))\n\n# Create dataset\ntrain_dataset = {\"damaged_sentence\": [], \"original_sentence\": []}\nfor data in tqdm(damaged_data):\n    train_dataset[\"damaged_sentence\"].append(data.texts[0])\n    train_dataset[\"original_sentence\"].append(data.texts[1])\ntrain_dataset = Dataset.from_dict(train_dataset)\n\n\ntrain_dataset[0]\n\n\n# # Choose a different deletion ratio\n# flat_sentences = list(set(flat_sentences))\n# damaged_data = DenoisingAutoEncoderDataset(\n#     flat_sentences,\n#     noise_fn=lambda s: DenoisingAutoEncoderDataset.delete(s, del_ratio=0.6)\n# )\n\n\nfrom sentence_transformers.evaluation import EmbeddingSimilarityEvaluator\n\n# Create an embedding similarity evaluator for stsb\nval_sts = load_dataset('glue', 'stsb', split='validation')\nevaluator = EmbeddingSimilarityEvaluator(\n    sentences1=val_sts[\"sentence1\"],\n    sentences2=val_sts[\"sentence2\"],\n    scores=[score/5 for score in val_sts[\"label\"]],\n    main_similarity=\"cosine\"\n)\n\n\nfrom sentence_transformers import models, SentenceTransformer\n\n# Create your embedding model\nword_embedding_model = models.Transformer('bert-base-uncased')\npooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), 'cls')\nembedding_model = SentenceTransformer(modules=[word_embedding_model, pooling_model])\n\n\nfrom sentence_transformers import losses\n\n# Use the denoising auto-encoder loss\ntrain_loss = losses.DenoisingAutoEncoderLoss(\n    embedding_model, tie_encoder_decoder=True\n)\ntrain_loss.decoder = train_loss.decoder.to(\"cuda\")\n\n\nfrom sentence_transformers.trainer import SentenceTransformerTrainer\nfrom sentence_transformers.training_args import SentenceTransformerTrainingArguments\n\n# Define the training arguments\nargs = SentenceTransformerTrainingArguments(\n    output_dir=\"tsdae_embedding_model\",\n    num_train_epochs=1,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=16,\n    warmup_steps=100,\n    fp16=True,\n    eval_steps=100,\n    logging_steps=100,\n)\n\n# Train model\ntrainer = SentenceTransformerTrainer(\n    model=embedding_model,\n    args=args,\n    train_dataset=train_dataset,\n    loss=train_loss,\n    evaluator=evaluator\n)\ntrainer.train()\n\n\n# Evaluate our trained model\nevaluator(embedding_model)\n\n\nimport gc\nimport torch\n\ngc.collect()\ntorch.cuda.empty_cache()",
    "crumbs": [
      "Home",
      "Training",
      "10. Embedding models"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "",
    "text": "from transformers import AutoTokenizer\nfrom datasets import load_dataset\n\n\n# Load a tokenizer to use its chat template\ntemplate_tokenizer = AutoTokenizer.from_pretrained(\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\")\n\ndef format_prompt(example):\n    \"\"\"Format the prompt to using the <|user|> template TinyLLama is using\"\"\"\n\n    # Format answers\n    chat = example[\"messages\"]\n    prompt = template_tokenizer.apply_chat_template(chat, tokenize=False)\n\n    return {\"text\": prompt}\n\n# Load and format the data using the template TinyLLama is using\ndataset = (\n    load_dataset(\"HuggingFaceH4/ultrachat_200k\",  split=\"test_sft\")\n      .shuffle(seed=42)\n      .select(range(3_000))\n)\ndataset = dataset.map(format_prompt)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Example of formatted prompt\nprint(dataset[\"text\"][2576])\n\n<|user|>\nGiven the text: Knock, knock. Who’s there? Hike.\nCan you continue the joke based on the given text material \"Knock, knock. Who’s there? Hike\"?</s>\n<|assistant|>\nSure! Knock, knock. Who's there? Hike. Hike who? Hike up your pants, it's cold outside!</s>\n<|user|>\nCan you tell me another knock-knock joke based on the same text material \"Knock, knock. Who's there? Hike\"?</s>\n<|assistant|>\nOf course! Knock, knock. Who's there? Hike. Hike who? Hike your way over here and let's go for a walk!</s>\n\n\n\n\n\n\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_name = \"TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\"\n\n# 4-bit quantization configuration - Q in QLoRA\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,  # Use 4-bit precision model loading\n    bnb_4bit_quant_type=\"nf4\",  # Quantization type\n    bnb_4bit_compute_dtype=\"float16\",  # Compute dtype\n    bnb_4bit_use_double_quant=True,  # Apply nested quantization\n)\n\n# Load the model to train on the GPU\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    device_map=\"auto\",\n\n    # Leave this out for regular SFT\n    quantization_config=bnb_config,\n)\nmodel.config.use_cache = False\nmodel.config.pretraining_tp = 1\n\n# Load LLaMA tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)\ntokenizer.pad_token = \"<PAD>\"\ntokenizer.padding_side = \"left\"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nfrom peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n\n# Prepare LoRA Configuration\npeft_config = LoraConfig(\n    lora_alpha=32,  # LoRA Scaling\n    lora_dropout=0.1,  # Dropout for LoRA Layers\n    r=64,  # Rank\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=  # Layers to target\n     ['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']\n)\n\n# prepare model for training\nmodel = prepare_model_for_kbit_training(model)\nmodel = get_peft_model(model, peft_config)\n\n\n\n\n\nfrom transformers import TrainingArguments\n\noutput_dir = \"./results\"\n\n# Training arguments\ntraining_arguments = TrainingArguments(\n    output_dir=output_dir,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=4,\n    optim=\"paged_adamw_32bit\",\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    num_train_epochs=1,\n    logging_steps=10,\n    fp16=True,\n    gradient_checkpointing=True\n)\n\n\n\n\n\n\nfrom trl import SFTTrainer\n\n# Set supervised fine-tuning parameters\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=dataset,\n    dataset_text_field=\"text\",\n    tokenizer=tokenizer,\n    args=training_arguments,\n    max_seq_length=512,\n\n    # Leave this out for regular SFT\n    peft_config=peft_config,\n)\n\n# Train model\ntrainer.train()\n\n# Save QLoRA weights\ntrainer.model.save_pretrained(\"TinyLlama-1.1B-qlora\")\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.\n\nDeprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.\n  warnings.warn(message, FutureWarning)\n/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1965: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n  warnings.warn(\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:397: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n  warnings.warn(\n\n\n\n    \n      \n      \n      [375/375 12:45, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n10\n1.670600\n\n\n20\n1.475400\n\n\n30\n1.451400\n\n\n40\n1.487800\n\n\n50\n1.477900\n\n\n60\n1.390500\n\n\n70\n1.495200\n\n\n80\n1.450300\n\n\n90\n1.427900\n\n\n100\n1.404400\n\n\n110\n1.414400\n\n\n120\n1.377500\n\n\n130\n1.332100\n\n\n140\n1.497000\n\n\n150\n1.347000\n\n\n160\n1.411500\n\n\n170\n1.454000\n\n\n180\n1.324500\n\n\n190\n1.419300\n\n\n200\n1.474900\n\n\n210\n1.404600\n\n\n220\n1.342100\n\n\n230\n1.361100\n\n\n240\n1.387300\n\n\n250\n1.353700\n\n\n260\n1.345800\n\n\n270\n1.465400\n\n\n280\n1.434000\n\n\n290\n1.387600\n\n\n300\n1.376200\n\n\n310\n1.395000\n\n\n320\n1.437900\n\n\n330\n1.387200\n\n\n340\n1.388100\n\n\n350\n1.313600\n\n\n360\n1.444300\n\n\n370\n1.452000\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n    \"TinyLlama-1.1B-qlora\",\n    low_cpu_mem_usage=True,\n    device_map=\"auto\",\n)\n\n# Merge LoRA and base model\nmerged_model = model.merge_and_unload()\n\n\n\n\n\nfrom transformers import pipeline\n\n# Use our predefined prompt template\nprompt = \"\"\"<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\n\"\"\"\n\n# Run our instruction-tuned model\npipe = pipeline(task=\"text-generation\", model=merged_model, tokenizer=tokenizer)\nprint(pipe(prompt)[0][\"generated_text\"])\n\n<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\nLarge Language Models (LLMs) are a type of artificial intelligence (AI) that can generate human-like language. They are trained on large amounts of data, including text, audio, and video, and are capable of generating complex and nuanced language.\n\nLLMs are used in a variety of applications, including natural language processing (NLP), machine translation, and chatbots. They can be used to generate text, speech, or images, and can be trained to understand different languages and dialects.\n\nOne of the most significant applications of LLMs is in the field of natural language generation (NLG). LLMs can be used to generate text in a variety of languages, including English, French, and German. They can also be used to generate speech, such as in chatbots or voice assistants.\n\nLLMs have the potential to revolutionize the way we communicate and interact with each other. They can help us create more engaging and personalized content, and they can also help us understand each other better.",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#data-preprocessing",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#data-preprocessing",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "",
    "text": "from transformers import AutoTokenizer\nfrom datasets import load_dataset\n\n\n# Load a tokenizer to use its chat template\ntemplate_tokenizer = AutoTokenizer.from_pretrained(\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\")\n\ndef format_prompt(example):\n    \"\"\"Format the prompt to using the <|user|> template TinyLLama is using\"\"\"\n\n    # Format answers\n    chat = example[\"messages\"]\n    prompt = template_tokenizer.apply_chat_template(chat, tokenize=False)\n\n    return {\"text\": prompt}\n\n# Load and format the data using the template TinyLLama is using\ndataset = (\n    load_dataset(\"HuggingFaceH4/ultrachat_200k\",  split=\"test_sft\")\n      .shuffle(seed=42)\n      .select(range(3_000))\n)\ndataset = dataset.map(format_prompt)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n# Example of formatted prompt\nprint(dataset[\"text\"][2576])\n\n<|user|>\nGiven the text: Knock, knock. Who’s there? Hike.\nCan you continue the joke based on the given text material \"Knock, knock. Who’s there? Hike\"?</s>\n<|assistant|>\nSure! Knock, knock. Who's there? Hike. Hike who? Hike up your pants, it's cold outside!</s>\n<|user|>\nCan you tell me another knock-knock joke based on the same text material \"Knock, knock. Who's there? Hike\"?</s>\n<|assistant|>\nOf course! Knock, knock. Who's there? Hike. Hike who? Hike your way over here and let's go for a walk!</s>",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#models---quantization",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#models---quantization",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "",
    "text": "import torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\nmodel_name = \"TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\"\n\n# 4-bit quantization configuration - Q in QLoRA\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,  # Use 4-bit precision model loading\n    bnb_4bit_quant_type=\"nf4\",  # Quantization type\n    bnb_4bit_compute_dtype=\"float16\",  # Compute dtype\n    bnb_4bit_use_double_quant=True,  # Apply nested quantization\n)\n\n# Load the model to train on the GPU\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    device_map=\"auto\",\n\n    # Leave this out for regular SFT\n    quantization_config=bnb_config,\n)\nmodel.config.use_cache = False\nmodel.config.pretraining_tp = 1\n\n# Load LLaMA tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)\ntokenizer.pad_token = \"<PAD>\"\ntokenizer.padding_side = \"left\"",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#configuration",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#configuration",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "",
    "text": "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n\n# Prepare LoRA Configuration\npeft_config = LoraConfig(\n    lora_alpha=32,  # LoRA Scaling\n    lora_dropout=0.1,  # Dropout for LoRA Layers\n    r=64,  # Rank\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=  # Layers to target\n     ['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']\n)\n\n# prepare model for training\nmodel = prepare_model_for_kbit_training(model)\nmodel = get_peft_model(model, peft_config)\n\n\n\n\n\nfrom transformers import TrainingArguments\n\noutput_dir = \"./results\"\n\n# Training arguments\ntraining_arguments = TrainingArguments(\n    output_dir=output_dir,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=4,\n    optim=\"paged_adamw_32bit\",\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    num_train_epochs=1,\n    logging_steps=10,\n    fp16=True,\n    gradient_checkpointing=True\n)",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#training",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#training",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "",
    "text": "from trl import SFTTrainer\n\n# Set supervised fine-tuning parameters\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=dataset,\n    dataset_text_field=\"text\",\n    tokenizer=tokenizer,\n    args=training_arguments,\n    max_seq_length=512,\n\n    # Leave this out for regular SFT\n    peft_config=peft_config,\n)\n\n# Train model\ntrainer.train()\n\n# Save QLoRA weights\ntrainer.model.save_pretrained(\"TinyLlama-1.1B-qlora\")\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': dataset_text_field, max_seq_length. Will not be supported from version '1.0.0'.\n\nDeprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.\n  warnings.warn(message, FutureWarning)\n/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1965: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a `max_seq_length` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a `dataset_text_field` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`.\n  warnings.warn(\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:397: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n  warnings.warn(\n\n\n\n    \n      \n      \n      [375/375 12:45, Epoch 1/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n10\n1.670600\n\n\n20\n1.475400\n\n\n30\n1.451400\n\n\n40\n1.487800\n\n\n50\n1.477900\n\n\n60\n1.390500\n\n\n70\n1.495200\n\n\n80\n1.450300\n\n\n90\n1.427900\n\n\n100\n1.404400\n\n\n110\n1.414400\n\n\n120\n1.377500\n\n\n130\n1.332100\n\n\n140\n1.497000\n\n\n150\n1.347000\n\n\n160\n1.411500\n\n\n170\n1.454000\n\n\n180\n1.324500\n\n\n190\n1.419300\n\n\n200\n1.474900\n\n\n210\n1.404600\n\n\n220\n1.342100\n\n\n230\n1.361100\n\n\n240\n1.387300\n\n\n250\n1.353700\n\n\n260\n1.345800\n\n\n270\n1.465400\n\n\n280\n1.434000\n\n\n290\n1.387600\n\n\n300\n1.376200\n\n\n310\n1.395000\n\n\n320\n1.437900\n\n\n330\n1.387200\n\n\n340\n1.388100\n\n\n350\n1.313600\n\n\n360\n1.444300\n\n\n370\n1.452000\n\n\n\n\n\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n  warnings.warn(\n\n\n\n\n\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n    \"TinyLlama-1.1B-qlora\",\n    low_cpu_mem_usage=True,\n    device_map=\"auto\",\n)\n\n# Merge LoRA and base model\nmerged_model = model.merge_and_unload()\n\n\n\n\n\nfrom transformers import pipeline\n\n# Use our predefined prompt template\nprompt = \"\"\"<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\n\"\"\"\n\n# Run our instruction-tuned model\npipe = pipeline(task=\"text-generation\", model=merged_model, tokenizer=tokenizer)\nprint(pipe(prompt)[0][\"generated_text\"])\n\n<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\nLarge Language Models (LLMs) are a type of artificial intelligence (AI) that can generate human-like language. They are trained on large amounts of data, including text, audio, and video, and are capable of generating complex and nuanced language.\n\nLLMs are used in a variety of applications, including natural language processing (NLP), machine translation, and chatbots. They can be used to generate text, speech, or images, and can be trained to understand different languages and dialects.\n\nOne of the most significant applications of LLMs is in the field of natural language generation (NLG). LLMs can be used to generate text in a variety of languages, including English, French, and German. They can also be used to generate speech, such as in chatbots or voice assistants.\n\nLLMs have the potential to revolutionize the way we communicate and interact with each other. They can help us create more engaging and personalized content, and they can also help us understand each other better.",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#data-preprocessing-1",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#data-preprocessing-1",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "Data Preprocessing",
    "text": "Data Preprocessing\n\nfrom datasets import load_dataset\n\ndef format_prompt(example):\n    \"\"\"Format the prompt to using the <|user|> template TinyLLama is using\"\"\"\n\n    # Format answers\n    system = \"<|system|>\\n\" + example['system'] + \"</s>\\n\"\n    prompt = \"<|user|>\\n\" + example['input'] + \"</s>\\n<|assistant|>\\n\"\n    chosen = example['chosen'] + \"</s>\\n\"\n    rejected = example['rejected'] + \"</s>\\n\"\n\n    return {\n        \"prompt\": system + prompt,\n        \"chosen\": chosen,\n        \"rejected\": rejected,\n    }\n\n# Apply formatting to the dataset and select relatively short answers\ndpo_dataset = load_dataset(\"argilla/distilabel-intel-orca-dpo-pairs\", split=\"train\")\ndpo_dataset = dpo_dataset.filter(\n    lambda r:\n        r[\"status\"] != \"tie\" and\n        r[\"chosen_score\"] >= 8 and\n        not r[\"in_gsm8k_train\"]\n)\ndpo_dataset = dpo_dataset.map(format_prompt, remove_columns=dpo_dataset.column_names)\ndpo_dataset\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nDataset({\n    features: ['chosen', 'rejected', 'prompt'],\n    num_rows: 5922\n})",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#models---quantization-1",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#models---quantization-1",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "Models - Quantization",
    "text": "Models - Quantization\n\nfrom peft import AutoPeftModelForCausalLM\nfrom transformers import BitsAndBytesConfig, AutoTokenizer\n\n# 4-bit quantization configuration - Q in QLoRA\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,  # Use 4-bit precision model loading\n    bnb_4bit_quant_type=\"nf4\",  # Quantization type\n    bnb_4bit_compute_dtype=\"float16\",  # Compute dtype\n    bnb_4bit_use_double_quant=True,  # Apply nested quantization\n)\n\n# Merge LoRA and base model\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n    \"TinyLlama-1.1B-qlora\",\n    low_cpu_mem_usage=True,\n    device_map=\"auto\",\n    quantization_config=bnb_config,\n)\nmerged_model = model.merge_and_unload()\n\n# Load LLaMA tokenizer\nmodel_name = \"TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\"\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)\ntokenizer.pad_token = \"<PAD>\"\ntokenizer.padding_side = \"left\"\n\n/usr/local/lib/python3.10/dist-packages/peft/tuners/lora/bnb.py:325: UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.\n  warnings.warn(",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#configuration-1",
    "href": "Hands-On-Large-Language-Models/chapter12/Chapter 12 - Fine-tuning Generation Models.html#configuration-1",
    "title": "Chapter 12 - Fine-tuning Generation Models",
    "section": "Configuration",
    "text": "Configuration\n\nfrom peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n\n# Prepare LoRA Configuration\npeft_config = LoraConfig(\n    lora_alpha=32,  # LoRA Scaling\n    lora_dropout=0.1,  # Dropout for LoRA Layers\n    r=64,  # Rank\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=  # Layers to target\n     ['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']\n)\n\n# prepare model for training\nmodel = prepare_model_for_kbit_training(model)\nmodel = get_peft_model(model, peft_config)\n\n\nfrom trl import DPOConfig\n\noutput_dir = \"./results\"\n\n# Training arguments\ntraining_arguments = DPOConfig(\n    output_dir=output_dir,\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=4,\n    optim=\"paged_adamw_32bit\",\n    learning_rate=1e-5,\n    lr_scheduler_type=\"cosine\",\n    max_steps=200,\n    logging_steps=10,\n    fp16=True,\n    gradient_checkpointing=True,\n    warmup_ratio=0.1\n)\n\n\nfrom trl import DPOTrainer\n\n# Create DPO trainer\ndpo_trainer = DPOTrainer(\n    model,\n    args=training_arguments,\n    train_dataset=dpo_dataset,\n    tokenizer=tokenizer,\n    peft_config=peft_config,\n    beta=0.1,\n    max_prompt_length=512,\n    max_length=512,\n)\n\n# Fine-tune model with DPO\ndpo_trainer.train()\n\n# Save adapter\ndpo_trainer.model.save_pretrained(\"TinyLlama-1.1B-dpo-qlora\")\n\n/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': max_prompt_length, max_length. Will not be supported from version '1.0.0'.\n\nDeprecated positional argument(s) used in DPOTrainer, please use the DPOConfig to set these arguments instead.\n  warnings.warn(message, FutureWarning)\n/usr/local/lib/python3.10/dist-packages/peft/tuners/lora/bnb.py:325: UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/dpo_trainer.py:358: UserWarning: You passed `max_length` to the DPOTrainer, the value you passed will override the one in the `DPOConfig`.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/dpo_trainer.py:371: UserWarning: You passed `max_prompt_length` to the DPOTrainer, the value you passed will override the one in the `DPOConfig`.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/trl/trainer/dpo_trainer.py:411: UserWarning: When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.\n  warnings.warn(\n\n\n\n\n\nmax_steps is given, it will override any value given in num_train_epochs\n/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n  warnings.warn(\n/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None\n  warnings.warn(\nCould not estimate the number of tokens of the input, floating-point operations will not be computed\n\n\n\n    \n      \n      \n      [200/200 12:52, Epoch 0/1]\n    \n    \n\n\n\nStep\nTraining Loss\n\n\n\n\n10\n0.692400\n\n\n20\n0.678200\n\n\n30\n0.646000\n\n\n40\n0.606300\n\n\n50\n0.595600\n\n\n60\n0.616800\n\n\n70\n0.593700\n\n\n80\n0.531900\n\n\n90\n0.559200\n\n\n100\n0.639000\n\n\n110\n0.496500\n\n\n120\n0.586000\n\n\n130\n0.630000\n\n\n140\n0.590100\n\n\n150\n0.577500\n\n\n160\n0.591000\n\n\n170\n0.606900\n\n\n180\n0.627800\n\n\n190\n0.668600\n\n\n200\n0.555400\n\n\n\n\n\n\n\nfrom peft import PeftModel\n\n# Merge LoRA and base model\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n    \"TinyLlama-1.1B-qlora\",\n    low_cpu_mem_usage=True,\n    device_map=\"auto\",\n)\nsft_model = model.merge_and_unload()\n\n# Merge DPO LoRA and SFT model\ndpo_model = PeftModel.from_pretrained(\n    sft_model,\n    \"TinyLlama-1.1B-dpo-qlora\",\n    device_map=\"auto\",\n)\ndpo_model = dpo_model.merge_and_unload()\n\n\nfrom transformers import pipeline\n\n# Use our predefined prompt template\nprompt = \"\"\"<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\n\"\"\"\n\n# Run our instruction-tuned model\npipe = pipeline(task=\"text-generation\", model=dpo_model, tokenizer=tokenizer)\nprint(pipe(prompt)[0][\"generated_text\"])\n\n<|user|>\nTell me something about Large Language Models.</s>\n<|assistant|>\nLarge Language Models (LLMs) are a type of artificial intelligence (AI) that can generate human-like language. They are trained on large amounts of data, including text, audio, and video, and are capable of generating complex and nuanced language.\n\nLLMs are used in a variety of applications, including natural language processing (NLP), machine translation, and chatbots. They can be used to generate text, speech, or images, and can be trained to understand different languages and dialects.\n\nOne of the most significant applications of LLMs is in the field of natural language generation (NLG). LLMs can be used to generate text in a variety of languages, including English, French, and German. They can also be used to generate speech, such as in chatbots or voice assistants.\n\nLLMs have the potential to revolutionize the way we communicate and interact with each other. They can help us create more engaging and personalized content, and they can also help us understand each other better.",
    "crumbs": [
      "Home",
      "Training",
      "12. Fine-tuning generation"
    ]
  },
  {
    "objectID": "Hands-On-Large-Language-Models/chapter14/Chapter 14 - Quantization and Local Inference.html",
    "href": "Hands-On-Large-Language-Models/chapter14/Chapter 14 - Quantization and Local Inference.html",
    "title": "Chapter 14 - Quantization and Local Inference",
    "section": "",
    "text": "This chapter connects GGUF quantization labels to concrete local behavior: file size, served metadata, and short deterministic completions. The examples use the Qwen 3.5 4B Q4 file because it leaves room for KV cache and repeated tests on a 12 GB GPU.\n\nfrom __future__ import annotations\n\nimport contextlib\nimport socket\nimport subprocess\nimport time\nfrom pathlib import Path\n\nimport pandas as pd\nimport requests\n\npd.set_option(\"display.max_colwidth\", 120)\n\nLLAMA_ROOT = Path.home() / \"tmp\" / \"llama-cpp\"\nBIN_DIR = LLAMA_ROOT / \"llama.cpp\" / \"build\" / \"bin\"\nMODEL_DIR = LLAMA_ROOT / \"models\"\nSERVER = BIN_DIR / \"llama-server\"\nBENCH = BIN_DIR / \"llama-bench\"\nQWEN_BF16 = MODEL_DIR / \"Qwen_Qwen3.5-4B-BF16.gguf\"\nQWEN_Q4 = MODEL_DIR / \"Qwen_Qwen3.5-4B-Q4_K_M.gguf\"\n\ndef gib(path: Path) -> float:\n    return path.stat().st_size / 1024**3\n\nsize_table = pd.DataFrame([\n    {\"file\": QWEN_BF16.name, \"format\": \"BF16\", \"size_gib\": round(gib(QWEN_BF16), 2)},\n    {\"file\": QWEN_Q4.name, \"format\": \"Q4_K_M\", \"size_gib\": round(gib(QWEN_Q4), 2)},\n])\ncompression = gib(QWEN_BF16) / gib(QWEN_Q4)\nprint(f\"BF16-to-Q4_K_M file-size ratio: {compression:.2f}x\")\nsize_table\n\nBF16-to-Q4_K_M file-size ratio: 2.80x\n\n\n\n\n\n\n\n\n\nfile\nformat\nsize_gib\n\n\n\n\n0\nQwen_Qwen3.5-4B-BF16.gguf\nBF16\n7.85\n\n\n1\nQwen_Qwen3.5-4B-Q4_K_M.gguf\nQ4_K_M\n2.81\n\n\n\n\n\n\n\n\ndef open_port() -> int:\n    with socket.socket() as sock:\n        sock.bind((\"127.0.0.1\", 0))\n        return sock.getsockname()[1]\n\ndef build_server_command(model: Path, alias: str, context: int, port: int, gpu_layers: str, flash_attn: str) -> list[str]:\n    return [\n        str(SERVER),\n        \"-m\", str(model),\n        \"-ngl\", gpu_layers,\n        \"-fa\", flash_attn,\n        \"-c\", str(context),\n        \"-np\", \"1\",\n        \"--cache-type-k\", \"q8_0\",\n        \"--cache-type-v\", \"q8_0\",\n        \"--jinja\",\n        \"--alias\", alias,\n        \"--host\", \"127.0.0.1\",\n        \"--port\", str(port),\n        \"--no-webui\",\n        \"--log-disable\",\n    ]\n\n@contextlib.contextmanager\ndef llama_server(model: Path, alias: str = \"qwen-local\", context: int = 1024):\n    attempts = [(\"full_gpu\", \"all\", \"on\"), (\"cpu_only\", \"0\", \"on\")]\n    errors = []\n    proc = None\n    try:\n        for mode, gpu_layers, flash_attn in attempts:\n            port = open_port()\n            command = build_server_command(model, alias, context, port, gpu_layers, flash_attn)\n            proc = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, text=True)\n            base_url = f\"http://127.0.0.1:{port}\"\n            deadline = time.time() + 90\n            ready = False\n            while time.time() < deadline:\n                if proc.poll() is not None:\n                    errors.append(f\"{mode} exited with code {proc.returncode}\")\n                    break\n                try:\n                    response = requests.get(f\"{base_url}/v1/models\", timeout=2)\n                    if response.ok:\n                        ready = True\n                        break\n                except requests.RequestException:\n                    time.sleep(1)\n            if ready:\n                print(f\"server_mode: {mode}\")\n                yield base_url, command, mode\n                return\n            if proc.poll() is None:\n                proc.terminate()\n                try:\n                    proc.wait(timeout=15)\n                except subprocess.TimeoutExpired:\n                    proc.kill()\n                    proc.wait(timeout=15)\n            proc = None\n        raise RuntimeError(\"llama-server did not become ready; \" + \"; \".join(errors))\n    finally:\n        if proc is not None and proc.poll() is None:\n            proc.terminate()\n            try:\n                proc.wait(timeout=15)\n            except subprocess.TimeoutExpired:\n                proc.kill()\n                proc.wait(timeout=15)\n\n\nwith llama_server(QWEN_Q4) as (base_url, command, mode):\n    model_response = requests.get(f\"{base_url}/v1/models\", timeout=10).json()\n    model_row = model_response[\"data\"][0]\n    metadata = model_row.get(\"meta\") or model_row.get(\"metadata\") or {}\n    selected = {\n        \"id\": model_row.get(\"id\"),\n        \"file_type\": metadata.get(\"general.file_type\") or metadata.get(\"ftype\"),\n        \"parameters\": metadata.get(\"general.parameter_count\") or metadata.get(\"n_params\"),\n        \"context\": metadata.get(\"llama.context_length\") or metadata.get(\"n_ctx\"),\n        \"embedding_length\": metadata.get(\"llama.embedding_length\") or metadata.get(\"n_embd\"),\n    }\n    print(\"Server command:\")\n    print(\" \".join(command))\n    pd.DataFrame([selected])\n\nserver_mode: cpu_only\nServer command:\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-server -m /home/alal/tmp/llama-cpp/models/Qwen_Qwen3.5-4B-Q4_K_M.gguf -ngl 0 -fa on -c 1024 -np 1 --cache-type-k q8_0 --cache-type-v q8_0 --jinja --alias qwen-local --host 127.0.0.1 --port 45667 --no-webui --log-disable\n\n\n\nprompts = [\n    \"Q: In one sentence, what does quantization buy us for local LLMs?\\nA:\",\n    \"Q: Name one risk of using an overly aggressive quantization.\\nA:\",\n    \"Q: Why is KV cache memory still important after quantizing weights?\\nA:\",\n]\n\nrows = []\nwith llama_server(QWEN_Q4) as (base_url, _, mode):\n    for prompt in prompts:\n        start = time.perf_counter()\n        response = requests.post(\n            f\"{base_url}/completion\",\n            json={\n                \"prompt\": prompt,\n                \"n_predict\": 64,\n                \"temperature\": 0,\n                \"stop\": [\"\\nQ:\"],\n            },\n            timeout=90,\n        )\n        elapsed = time.perf_counter() - start\n        response.raise_for_status()\n        payload = response.json()\n        timings = payload.get(\"timings\", {})\n        rows.append({\n            \"prompt\": prompt.split(\"\\n\")[0],\n            \"answer\": payload.get(\"content\", \"\").strip(),\n            \"elapsed_s\": round(elapsed, 2),\n            \"predicted_tokens\": timings.get(\"predicted_n\"),\n            \"predicted_tokens_per_s\": round(timings.get(\"predicted_per_second\", 0.0), 2),\n        })\n\ninference_results = pd.DataFrame(rows)\ninference_results\n\nserver_mode: cpu_only\n\n\n\n\n\n\n\n\n\nprompt\nanswer\nelapsed_s\npredicted_tokens\npredicted_tokens_per_s\n\n\n\n\n0\nQ: In one sentence, what does quantization buy us for local LLMs?\nQuantization reduces the model size and memory footprint, enabling efficient inference on devices with limited resou...\n4.91\n22\n5.92\n\n\n1\nQ: Name one risk of using an overly aggressive quantization.\n<think>\\n\\n</think>\\n\\nOne significant risk of using an overly aggressive quantization is **catastrophic performance...\n11.70\n64\n5.97\n\n\n2\nQ: Why is KV cache memory still important after quantizing weights?\nBecause quantizing weights only reduces the memory footprint of the model parameters, but the KV cache is still need...\n6.60\n32\n5.82\n\n\n\n\n\n\n\n\nbench_command = [\n    str(BENCH),\n    \"-m\", str(QWEN_Q4),\n    \"-ngl\", \"all\",\n    \"-fa\", \"1\",\n    \"-c\", \"2048\",\n    \"-n\", \"32\",\n]\n\ntry:\n    bench = subprocess.run(bench_command, capture_output=True, text=True, timeout=45)\n    print(f\"llama-bench return code: {bench.returncode}\")\n    text = (bench.stdout or bench.stderr).strip()\n    print(\"\\n\".join(text.splitlines()[:20]))\nexcept subprocess.TimeoutExpired:\n    print(\"llama-bench did not finish within the timeout.\")\n\nllama-bench return code: 127\n/home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-bench: symbol lookup error: /home/alal/tmp/llama-cpp/llama.cpp/build/bin/llama-bench: undefined symbol: llama_memory_breakdown_print\n\n\nThe served Q4 model is small enough to run repeated deterministic requests comfortably. The benchmark command is included as a stack diagnostic: if it fails while llama-server works, the immediate problem is the benchmark binary or runtime linkage rather than model availability.\n\n\n\n Back to top",
    "crumbs": [
      "Home",
      "Consumer Hardware",
      "14. Quantization and inference"
    ]
  },
  {
    "objectID": "index.html",
    "href": "index.html",
    "title": "Hands-On Large Language Models",
    "section": "",
    "text": "Introduction to Language Models\nTokens and Token Embeddings\nLooking Inside LLMs\nText Classification\nText Clustering and Topic Modeling\nPrompt Engineering\nAdvanced Text Generation Techniques and Tools\nSemantic Search\nMultimodal Large Language Models\nCreating Text Embedding Models\nFine-Tuning BERT\nFine-tuning Generation Models",
    "crumbs": [
      "Home",
      "Overview"
    ]
  },
  {
    "objectID": "index.html#chapters",
    "href": "index.html#chapters",
    "title": "Hands-On Large Language Models",
    "section": "",
    "text": "Introduction to Language Models\nTokens and Token Embeddings\nLooking Inside LLMs\nText Classification\nText Clustering and Topic Modeling\nPrompt Engineering\nAdvanced Text Generation Techniques and Tools\nSemantic Search\nMultimodal Large Language Models\nCreating Text Embedding Models\nFine-Tuning BERT\nFine-tuning Generation Models",
    "crumbs": [
      "Home",
      "Overview"
    ]
  },
  {
    "objectID": "index.html#consumer-hardware-follow-up",
    "href": "index.html#consumer-hardware-follow-up",
    "title": "Hands-On Large Language Models",
    "section": "Consumer-Hardware Follow-up",
    "text": "Consumer-Hardware Follow-up\n\nCurriculum plan\nChapter 13: The Local Model Stack\nChapter 14: Quantization and Local Inference\nChapter 15: Serving Models Locally",
    "crumbs": [
      "Home",
      "Overview"
    ]
  }
]