andyzoujm / representation-engineering Goto Github PK
View Code? Open in Web Editor NEWRepresentation Engineering: A Top-Down Approach to AI Transparency
Home Page: https://www.ai-transparency.org/
License: MIT License
Representation Engineering: A Top-Down Approach to AI Transparency
Home Page: https://www.ai-transparency.org/
License: MIT License
Hi team,
Thank you for your great project! I am recently running your honesty example notebook and trying to run through your provided example. However in the last two cells of honesty.ipynb. I got no meaningful output from the control side. I made no adjustments to the code and used the 7B model. Any ideas would be helpful!
Thanks
Hey lab. I am working on a POC with a customer support AI company. They ask us to provide honesty benchmark to use data to prove we are more honest than the vanilla llama2. Do we have the public data set and the test result? If not, could we use https://people.eecs.berkeley.edu/~normanmu/llm_rules/ to test? It seems the pipeline is different and we can not use as is.
We realized we have to manually adjust the [threshold] parameter in Honest example in order to get acceptable results. We want to know what is the plan to automate the [threshold] parameter adjustment? It is not very user friendly and not very practical as well.
Hi
In your honest scores calculation, what is the justification of
results[pos][0][layer][0] * honesty_rep_reader.direction_signs[layer][0]
Why you need to multiply by the direction sign, not just using the results[pos][0][layer][0]
Thanks
As far as I can tell, this is the part where the Contrast Vector is applied:
representation-engineering/repe/rep_control_contrast_vec.py
Lines 331 to 338 in 8de198b
I asked myself the question why not only the hidden_states
are changed but also hidden_states_p
and hidden_states_n
. Could you elaborate on that?
Btw: Since contrast_tokens
is choosen to be negtative, is it not always equal to c_length
?
Hi,
I was wondering if code demonstrating how to use the contrast vector for text generation could be pushed? (Ideally for a 7B model!)
I am quite unclear on how to convert the contrast vector code from honesty_control_TQA.ipynb to allow for generation. And I am finding it difficult to parse all the contrast vector implementation details from the paper.
This would be very useful, so would really appreciate it!
Thanks,
Robert
Hi! When I run your notebook, I wonder how can define layer_id for emotion control?
Looking forward to hear your answers, thank you!
Hello authors,
Your experiment results on harmfulness classification:https://github.com/andyzoujm/representation-engineering/blob/main/examples/harmless_harmful/harmless_llama2.ipynb
shows that Llama-2-13b-chat achieves near 100% acc, even in the lower layers.
I have tried more model: Llama-2-{7,70}b-chat, llama-2-7b, bloomz-{560m,1b1,1b7,3b,7b1}, bloom-7b1, all of these models also achieves near perfect performance.
I think the phenomenon is trivial for chat models because harmful and harmless input correspond to the two directions of refusing to answer and answering, however, for models without alignment like bloom/llama-2, it is nontrivial.
(By the way, in my experiment, I didn't include task template in prompt, instead, i directly used stimuli, and I used the ClusterMeanRepReader method.)
Are there any more plausible explanations for the phenomenon?
Hi! Thanks for maintaining the codebase with thorough documentation and comprehensive examples! Is it possible to add the CLIP examples in Appendix B.5? Also, could you clarify how the emotion directions are obtained? Is it obtained with texts and then used to classify images?
Thanks a lot in advance!
Hi! When I run examples/honesty/honesty_control_TQA.ipynb with llama2-7b-chat-hf, it threw an assert error in line 44:
assert np.isnan(output_logprobs).sum() == 0, "NaN in output logprobs"
I already set layer_ids = np.arange(8, 32, 3).
Wish to get your help.
Hello~, I'm reading the paper and want to do some tests by myself, will the related code and data be released in this repo?
How should I apply emotion control in a multi-turn conversation if I want to use it?
Hi,
I am wondering why training dataset in honest reading notebook have unfinished sentences in the inputs and labels containing [True, False]. How are inputs and labels connected?
Thanks
In Algorithm 1, we optimized over the loss
While in example code, we never generate from the constrastive perturbations directly,
https://github.com/andyzoujm/representation-engineering/blob/main/examples/honesty/honesty_contrast_vec_TQA_generation.ipynb
Do we need to optimize over the Loss L in Algorithm 1 to reproduce the results?
Hello, author, how can I generate dishonest responses?
Hi, I have some questions about the emotion_function notebook.
Thanks for your help!
Hi,
What is the purpose of this in your rep_reader.py?
pca_outputs_min = np.mean([o[train_labels[i].index(1)] == min(o) for i, o in enumerate(pca_outputs_comp)])
pca_outputs_max = np.mean([o[train_labels[i].index(1)] == max(o) for i, o in enumerate(pca_outputs_comp)])
layer_signs[component_index] = np.sign(np.mean(pca_outputs_max) - np.mean(pca_outputs_min))
Then in your inference code on testing data
Why do you evaluate like this:
sign = rep_reader.direction_signs[layer][component_index]
eval_func = min if sign == -1 else max
cors = np.mean([eval_func(H) == H[0] for H in H_test])
It is not clearly stated in the paper as well.
Thanks!
I would like to ask the author, in Step 1: Designing Stimulus and Task of LAT USER: <experimental/reference prompt>
ASSISTANT: Are the and of this template used as stimulus input, or are used as input and used as output? We shall designate this template for a function f as Tf+
when using the experimental prompt and Tf−
when using the reference prompt. What is the function f here? Could you please ask the author to help me answer my doubts? Thank you!
Andy and the team:
We made two performance enhancements: Flash Attention & Int8 quantization to be able to make the execution speed 4-5 times faster. Please let us know if we are allowed to contribute the source code back to the community.
Regards
Founder of ReparteeAI
Danny
Forgive my English expression ability, I would be appreciated for ur answer and it would be super if u provide a schedule of code updating.
We see huge demand for text-to-image honesty check. Please help us provide such capability
Currently, after training the rep_reader
, the coeff
variable used in the control pipeline need to be customized solely by experiment, and the value changes a lot, take the primary_emotions
as a example, here's the values I found:
# LLaMA-2-Chat-13B coeff=3.0-3.5
# mistralai/Mistral-7B-Instruct-v0.1 coeff=0.5
# HuggingFaceH4/zephyr-7b-beta coeff=0.3
# openchat/openchat_3.5 coeff=0.2
This makes it challenging for RepControl to adapt to new models.
My finding is that by introducing the pca_model's explained_variance_ratio_
into control progress can make the manipulation progress more "gentle" / "accurate".
Here's the key modifications:
In the rep_readers.py :
def get_rep_directions(self, model, tokenizer, hidden_states, hidden_layers, **kwargs):
"""Get PCA components for each layer"""
directions = {}
# like directions, save the variance ratio for each layer
variance_ratio = {}
for layer in hidden_layers:
........
self.n_components = pca_model.n_components_
variance_ratio[layer] = pca_model.explained_variance_ratio_
self.variance_ratio = variance_ratio
return directions
Each layer's variance_ratio represents how sparse or variably distributed the direction is, which can be interpreted as a 'confidence' score in the control section for that layer.
So, when manipulating the output, the activation
variable is calculated as:
coeff=0.2
coeff_with_variance = 2.0
activations = {}
activations_with_variance = {}
for layer in layer_id:
activations[layer] = torch.tensor(coeff * rep_reader.directions[layer] * rep_reader.direction_signs[layer]).to(model.device).half()
variance_ratio = rep_reader.variance_ratio[layer][0]
# print(variance_ratio)
activations_with_variance[layer] = torch.tensor(coeff_with_variance * rep_reader.directions[layer] * rep_reader.direction_signs[layer] * variance_ratio).to(model.device).half()
Applying this method seems to allow all the 7B models I've tested to adapt a common coeff value, approximately around 2.0.
Theoretically, I came up with this idea when I saw the code of WrappedBlock using the controller (activations) to manipulate the tensor using a simple linear approach.
So, I just take the variance_ratio into account in a most simple way.
Maybe, by extracting the PCA model's underlying singular vector can gain better control over this.
Thanks for sharing this great work!
I notice in the paper and the example Jupyter code, the output of ASSISTANT(response), or the statement is truncated, I would like to know the reason. Thank you so much!
I am confused with the code "Contrast Vector Control" in honesty_control_TQA.ipynb.
You use the correct answer as input along with the question to get your direction vectors. Doesn't this constitute data leakage?
I don't know if I didn't fully understand the code. Please help me to solve this problem.
inputs_neg_s, masks_neg_s, split_neg = prepare_decoder_only_inputs(q_batch_neg, a_batch, tokenizer, model.model.device)
split = inputs_neg_s['input_ids'].shape[1] - split_neg
for layer_id in layer_ids:
with torch.no_grad():
_ = wrapped_model(**inputs_pos_s)
pos_outputs = wrapped_model.get_activations(layer_ids, block_name=block_name)
_ = wrapped_model(**inputs_neg_s)
neg_outputs = wrapped_model.get_activations(layer_ids, block_name=block_name)
directions[layer_id] += coeff * (pos_outputs[layer_id][:, -split:] - neg_outputs[layer_id][:, -split:]) / len(templates)
wrapped_model.reset()
wrapped_model.set_controller([l for l in layer_ids if l <= layer_id], directions,
masks=masks[:, -split:, None],
token_pos="end",
normalize=False)
Hi, thanks for your great contribution.
I have meet some questions when I try to use llama2-7b-chat-hf on the honesty.ipynb
when I load the model with a dtype of bfloat 16, it shows the hidden layer has nan of inf data so the program goes wrong when doing the pca.
when I load the model with a dtype of float 32, it shows the data is nan of inf and the program goes wrong when doing the generation test.
Hoping to get your help!
I see your attempts trying to only align the CoT side.
Since the 'ehartford/Wizard-Vicuna-30B-Uncensored' model file is approximately over 120GB, does it mean that my computer's RAM memory also needs to be greater than 120GB to load it? If so, are there any smaller models you would recommend?
true_statements = df[df['label'] == 1]['statement'].values.tolist()
false_statements = df[df['label'] == 0]['statement'].values.tolist()
template_str = "Pretend you're {type} person making statements about the world."
honest_statements = []
untruthful_statements = []
# Process statements
for statement in true_statements:
tokens = tokenizer.tokenize(statement)
for idx in range(1, len(tokens) - 5):
truncated_tokens = tokens[:idx]
truncated_statement = tokenizer.convert_tokens_to_string(truncated_tokens)
honest_statements.append(f"{user_tag} {template_str.format(type='an honest')} {assistant_tag} " + truncated_statement)
untruthful_statements.append(f"{user_tag} {template_str.format(type='an untruthful')} {assistant_tag} " + truncated_statement)
false_statements not being used?
false_statements
Hi, I wonder if the performance of your LoRRA models is shown in Open LLM Leaderboard? Thanks!
Hi authors,
I have some questions about Harmlessness concept and controlling:
I would to know how to construct stimulus and how to get hidden states. You mentioned you adopt the below format:
Therefore, only the instruction part varies, depending on the type of instruction (harmful or harmless). Addtionally, you also get the hidden states of final token ([/INST] or ASSISTANT). Do I understand correctly?
You mentioned that you also utilize harmless instructions from ShareGPT and construct contrast pairs to obtain more Significant activations. When predicting whether a specific input sample is harmful or harmless, do we still need to use a contrast pair instead of just a single input? like precedures shown in https://github.com/andyzoujm/representation-engineering/tree/main/examples/honesty
In Section 4.1, it mentions that there are 3 ways to obtain stimulus for truthfulness.
(1) Fifty examples from the ARC-Challenge training set, (2) Five examples generated by the LLaMA-2-Chat-13B model in response to requests for question-answer pairs with varying degrees of truthfulness, (3) The six QA primer examples used in the original implementation, each of which is paired with a false answer generated by LLaMA-2-Chat-13B.
And I wonder in the second and third way, how do you prompt Llama2-13b-chat to generate false answers?
I have been experimenting with the linear, piece-wise, and projection operations for representation control. It would be useful to have the projection operation available for reference to make sure my implementation matches what was used in the paper. But, it's currently not implemented.
representation-engineering/repe/rep_control_reading_vec.py
Lines 93 to 95 in 8de198b
I found the calculation of the reading pipeline is super slow. I send the projection and recenter to GPU and make it run 20x faster. May I open a pull request?
def project_onto_direction(H, direction):
"""Project matrix H (n, d_1) onto direction vector (d_2,)"""
# Calculate the magnitude of the direction vector
# Ensure H and direction are on the same device (CPU or GPU)
device = H.device
if type(direction) != torch.Tensor:
direction = torch.Tensor(direction)
direction = direction.to(device)
mag = torch.norm(direction)
assert not torch.isinf(mag).any()
# Calculate the projection
projection = H.matmul(direction) / mag
return projection
def recenter(x, mean=None):
if mean is None:
# mean = x.mean(axis=0, keepdims=True)
mean = torch.mean(x,axis=0,keepdims=True)
else:
mean = torch.Tensor(mean).cuda()
# print(type(x), type(mean))
return x - mean
Hi, Thank you for making the code directly available!
I have a question about the clustermean method. The code (repe.rep_reading_pipeline
, l. 106) contains the below statement which I think is not effective since it's 'cluster_mean'
. However, independently of that, since I am trying to just understand, shouldn't it be n_difference == 0
?
if direction_method == 'clustermean':
assert n_difference == 1, "n_difference must be 1 for clustermean"
Thank you so much already!
We are working on a POC that requires RAG. We found Repe using HF transformer custom pipeline but langchain(HF using) does not support transformer custom pipeline. Any suggestion?
I am not able to train larger model on two GPUs, does anyone know how to fix this with deepspeed?
I tried to use llama2_lorra.py with the script llama_lorra_tqa_7b.sh with 4 GPUs. Though I know --num_gpus=1 fixed the issue.
ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode. Please rerun your script specifying `--num_processes=1` or by launching with `python {{myscript.py}}`.
{'tqa_accuracy': 0.31334149326805383, 'arc-e_accuracy': 0.6614035087719298}
deepspeed --master_port $ds_master_port --num_gpus=1 src/llama2_lorra.py
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.