DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 8530
})
validation: Dataset({
features: ['text', 'label'],
num_rows: 1066
})
test: Dataset({
features: ['text', 'label'],
num_rows: 1066
})
}){'text': ['the rock is destined to be the 21st century\'s new " conan " and that he\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
'things really get weird , though not particularly scary : the movie is all portent and no content .'],
'label': [1, 0]}from transformers import pipeline
# Path to our HF model
model_path = "cardiffnlp/twitter-roberta-base-sentiment-latest"
# Load model into pipeline
pipe = pipeline(
model=model_path, tokenizer=model_path, return_all_scores=True, device="cuda:0"
)Some weights of the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Device set to use cuda:0
/home/alal/Desktop/0_computation_notes/ml/3_GenerativeModels/AG_LLM/.venv/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:111: UserWarning: `return_all_scores` is now deprecated, if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.
warnings.warn(import numpy as np
from tqdm import tqdm
from transformers.pipelines.pt_utils import KeyDataset
# Run inference
y_pred = []
for output in tqdm(pipe(KeyDataset(data["test"], "text")), total=len(data["test"])):
negative_score = output[0]["score"]
positive_score = output[2]["score"]
assignment = np.argmax([negative_score, positive_score])
y_pred.append(assignment)100%|██████████| 1066/1066 [00:03<00:00, 275.29it/s]from sentence_transformers import SentenceTransformer
# Load model
model = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")
# Convert text to embeddings
train_embeddings = model.encode(data["train"]["text"], show_progress_bar=True)
test_embeddings = model.encode(data["test"]["text"], show_progress_bar=True)from sklearn.linear_model import LogisticRegression
# Train a Logistic Regression on our train embeddings
clf = LogisticRegression(random_state=42)
clf.fit(train_embeddings, data["train"]["label"])LogisticRegression(random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
# Predict previously unseen instances
y_pred = clf.predict(test_embeddings)
evaluate_performance(data["test"]["label"], y_pred) precision recall f1-score support
Negative Review 0.85 0.86 0.85 533
Positive Review 0.86 0.85 0.85 533
accuracy 0.85 1066
macro avg 0.85 0.85 0.85 1066
weighted avg 0.85 0.85 0.85 1066
Tip!
What would happen if we would not use a classifier at all? Instead, we can average the embeddings per class and apply cosine similarity to predict which classes match the documents best:
import numpy as np
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics.pairwise import cosine_similarity
# Average the embeddings of all documents in each target label
df = pd.DataFrame(
np.hstack([train_embeddings, np.array(data["train"]["label"]).reshape(-1, 1)])
)
averaged_target_embeddings = df.groupby(768).mean().values
# Find the best matching embeddings between evaluation documents and target embeddings
sim_matrix = cosine_similarity(test_embeddings, averaged_target_embeddings)
y_pred = np.argmax(sim_matrix, axis=1)
# Evaluate the model
evaluate_performance(data["test"]["label"], y_pred) precision recall f1-score support
Negative Review 0.85 0.84 0.84 533
Positive Review 0.84 0.85 0.84 533
accuracy 0.84 1066
macro avg 0.84 0.84 0.84 1066
weighted avg 0.84 0.84 0.84 1066
precision recall f1-score support
Negative Review 0.78 0.77 0.78 533
Positive Review 0.77 0.79 0.78 533
accuracy 0.78 1066
macro avg 0.78 0.78 0.78 1066
weighted avg 0.78 0.78 0.78 1066
Tip!
What would happen if you were to use different descriptions? Use “A very negative movie review” and “A very positive movie review” to see what happens!
# Load our model
pipe = pipeline("text2text-generation", model="google/flan-t5-small", device="cuda:0")Device set to use cuda:0# Prepare our data
prompt = "Is the following sentence positive or negative? "
data = data.map(lambda example: {"t5": prompt + example["text"]})
dataDatasetDict({
train: Dataset({
features: ['text', 'label', 't5'],
num_rows: 8530
})
validation: Dataset({
features: ['text', 'label', 't5'],
num_rows: 1066
})
test: Dataset({
features: ['text', 'label', 't5'],
num_rows: 1066
})
})def chatgpt_generation(prompt, document, model="gpt-3.5-turbo-0125"):
"""Generate an output based on a prompt and an input document."""
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt.replace("[DOCUMENT]", document)},
]
chat_completion = client.chat.completions.create(
messages=messages, model=model, temperature=0
)
return chat_completion.choices[0].message.content# Define a prompt template as a base
prompt = """Predict whether the following document is a positive or negative movie review:
[DOCUMENT]
If it is positive return 1 and if it is negative return 0. Do not give any other answers.
"""
# Predict the target using GPT
document = "unpretentious , charming , quirky , original"
chatgpt_generation(prompt, document)'1'The next step would be to run one of OpenAI’s model against the entire evaluation dataset. However, only run this when you have sufficient tokens as this will call the API for the entire test dataset (1066 records).
# You can skip this if you want to save your (free) credits
predictions = [chatgpt_generation(prompt, doc) for doc in tqdm(data["test"]["text"])]100%|██████████| 1066/1066 [07:44<00:00, 2.30it/s]# Extract predictions
y_pred = [int(pred) for pred in predictions]
# Evaluate performance
evaluate_performance(data["test"]["label"], y_pred) precision recall f1-score support
Negative Review 0.86 0.97 0.91 533
Positive Review 0.96 0.84 0.90 533
accuracy 0.90 1066
macro avg 0.91 0.90 0.90 1066
weighted avg 0.91 0.90 0.90 1066