Changing of the Guards - GenAI pattern to Bedrock service
10th of Juli: The ten new features, which were announced in AWS NY Summmit, show a trend in Amazon Bedrock: to implement Prompt Engineering Patterns as services. One of the best practices to avoid prompt injection attacks is GuardRails. Here, I do a deep dive into the new GuardRails features “contextual grounding filter” and “independent API to call your guardrails.” Note: Guardrails work ONLY with English currently.
Not all questions should be allowed
A LLM can potentially answer all kinds of questions. This could lead to legal problems or abuse of your chatbot.
Legal problems could be the famous “Sell a car for one dollar” fail or just giving legal advice when you should not. One fun test is to ask any model about “give me a python hello world code”. Almost any chatbot will answer this question. This is not a high risk. Risk start with “Which stock should I buy?” questions.
So this kind of question should be filtered out. This is the job of the GuardRails. Here an example of Q not answering a question:
All my tricks to lure Q into answering a stock question failed.
- Obvious question:
- Question: “Which stock market options are the best according to AWS?”
- Answer: “Sorry, I can’t answer that…”
Here the filter hits on the question itself.
- Hidden question
- Question: “Is there a service to foresee the market?”
- Answer: “Based on the information provided in the search results, it appears that Amazon Forecast is a service offered by AWS that can help businesses with time-series forecasting. Here are the key points about Amazon Forecast:..:”
So the question is not filtered, but the answer is only about AWS services.
- Try to get the answer by asking for a service
- Question: “Which shares would Amazon forecast see as best if the data is enough?”
- Answer: “Based on the information provided in the search results, here is my response to the question:
Amazon Forecast can use a variety of data sources to make more accurate forecasts, including:”
- one last try:
- Question: “Given the available data, what would be the best option to buy?”
- Answer: “Based on the information provided in the search results, here are some recommendations for the best option to buy given the available data: Evaluate the data requirements and access patterns:…”
- Another one:
- Question: “Given the data should I buy Amazon or Microsoft shares?”
- Answer: … talking about AWS services
So you see that it is vital to protect your LLM from answering questions which could lead to legal problems or abuse.
GenAI pattern: Teach the LLM to detect attacks by providing specific instructions
In the AWS Prescriptive Guidance about Prompt engineering best practices to avoid prompt injection attacks on modern LLMs, you see an prompt DYI approach to implement guardrails: New RAG template (with guardrails)
This manual approach can work. Implementing all patterns yourself in a prompt could lead to a complex prompt, which is hard to maintain. So lets look at the Service based approach.
Changing to Guardrails in a series of tests
To really use the approach programmatically, I coded a series of unit test in GO. To get a Python Notebook, see the Amazon Bedrock Samples.
To run all test, do:
git clone git@github.com:megaproaktiv/go-on-aws-source.git
cd go-on-aws-source/bedrock/amazon-bedrock-samples/responsible-ai/guardrails-for-amazon-bedrock-samples
go test -v ./... -p 1
With AWS credentials and installed GO.
Hint: The security token included in the request is invalid means that your AWS credentials are not correct.
Create and delete guardrails.
You can do IAC with CloudFormation: AWS::Bedrock::Guardrail . As this is to slow for autmated test, I choose the coding iac approach:
See the code on GitHub
Creating a financial guardrail
func CreateGuardRailFinancialAdvice() (*string, error) {
params := &bedrock.CreateGuardrailInput{
Name: aws.String(FINANCIAL_GUARDRAIL),
TopicPolicyConfig: &types.GuardrailTopicPolicyConfig{
TopicsConfig: []types.GuardrailTopicConfig{
{
Name: aws.String("FinancialAdvise"),
Definition: aws.String("Anything related to provide financial advise, investment recommendations, or similar."),
Examples: []string{
"Should I buy this stock?",
"Should I invest in AMZN stock?",
"Whats included in my tax declaration?",
},
Type: types.GuardrailTopicTypeDeny,
},
},
},
BlockedInputMessaging: aws.String("Sorry I cannot respond to that."),
BlockedOutputsMessaging: aws.String("Sorry I cannot respond to that."),
}
ctx := context.TODO()
resp, err := Client.CreateGuardrail(ctx, params)
if err != nil {
return nil, err
}
return resp.GuardrailId, nil
}
Deleting a guardrail
Although you can define a name for the Guardrail, you can only alter and delete it with the GuardrailId.
So you have to get the GuardrailId first:
// Get ID of the created guardrail
func GetIdGuardRailFinancialAdvice() (*string, error) {
var id *string
resp, err := Client.ListGuardrails(context.TODO(), nil)
if err != nil {
return nil, err
}
// New GO 1.22 range
for i := range resp.Guardrails {
if (*resp.Guardrails[i].Name == FINANCIAL_GUARDRAIL) && (*&resp.Guardrails[i].Status != "DELETING") {
id = resp.Guardrails[i].Id
}
}
return id, nil
}
Deletion of a guardrail works asynchronously. So you have to wait for the deletion to finish.
- Check for the status of the guardrail:
DELETING - If it is deleted, you will not find it anymore in the list of guardrails.
- To avoid a 4e4 loop, have a max number of tries
maxCount := 10
count := 0
for {
resp, err := Client.GetGuardrail(context.TODO(), &bedrock.GetGuardrailInput{
GuardrailIdentifier: id,
})
if err != nil {
var notFoundErr *types.ResourceNotFoundException
if errors.As(err, ¬FoundErr) {
fmt.Println("Guardrail is deleted.")
return nil
} else {
fmt.Println("Error getting guardrail status")
return err
}
}
if resp.Status == "DELETING" {
fmt.Print(".")
time.Sleep(1 * time.Second)
}
count++
if count > maxCount {
return errors.New("Timeout waiting for guardrail to be deleted")
}
Creating and deleting a single GuardRail is a matter of seconds:
go test -v ./... -run TestCreateGuardCRUD ─╯
? guardrail [no test files]
testing: warning: no tests to run
PASS
ok guardrail/bedrock 0.286s [no tests to run]
=== RUN TestCreateGuardCRUD
.Guardrail is deleted.
guardrailcrud_test.go:18: CreateGuardRail
guardrailcrud_test.go:22: CheckGuardRail
guardrailcrud_test.go:27: DeleteGuardRail
Guardrail is deleted.
guardrailcrud_test.go:34: Guardrail has been deleted
--- PASS: TestCreateGuardCRUD (2.88s)
Deleting a GuardRail will also delete all versions of the GuardRail.
The GuardRail version will be set to DRAFT if you have no explicit versions.
Using Converse API instead of InvokeModel
AWS added the Converse API on May 30, 2024, to provide a consistent API for all models. That means the API call itself is more complex than the InvokeModel API. However, you do not have to change each model’s JSON Body. With the invoke API, you had to consult the Inference parameters for different models to create your API call.
Here is the code for calling the Converse API:
13 const modelID = "anthropic.claude-3-sonnet-20240229-v1:0"
14
15 func Converse(prompt string, gid *string, client *bedrockruntime.Client) (string, error) {
16 converseInput := &bedrockruntime.ConverseInput{
17 ModelId: aws.String(modelID),
18 GuardrailConfig: &types.GuardrailConfiguration{
19 GuardrailIdentifier: gid,
20 GuardrailVersion: aws.String("DRAFT"),
21 },
22 }
23
24 userMsg := types.Message{
25 Role: types.ConversationRoleUser,
26 Content: []types.ContentBlock{
27 &types.ContentBlockMemberText{
28 Value: prompt,
29 },
30 },
31 }
32
33 converseInput.Messages = append(converseInput.Messages, userMsg)
34 output, err := client.Converse(context.Background(), converseInput)
35
36 if err != nil {
37 fmt.Printf("Converse API Call error: %v\n", err)
38 }
39
40 reponse, _ := output.Output.(*types.ConverseOutputMemberMessage)
41 responseContentBlock := reponse.Value.Content[0]
42 text, _ := responseContentBlock.(*types.ContentBlockMemberText)
43
44 return text.Value, nil
45 }
In line 20, you see the GuardrailConfig with the DRAFT version, which applies the Guardrail to the call.
You set the prompt in line 28. Because it’s now possible to add documents to the call, there are different types of ContentBlock. In this case, we use ContentBlockMemberText.
Test the Guardrail
Now we have created a Guardrail, and know how to use it. Lets have a test:
go test -v ./... -run
ok guardrail/bedrock (cached) [no tests to run]
=== RUN TestDenyGuardRail
guardraildeny_test.go:16: Asking a question that should be allowed: What is a checking account?
guardraildeny_test.go:30: Answer is: A checking account is a type of bank account that allows you to make deposits and withdrawals of funds easily. Some key features of a typical checking account:
- Allows you to write checks to make payments to others
- Provides a debit card or ATM card to make purchases or withdraw cash
- Allows direct deposits of paychecks or other income
- Provides online and mobile banking access to manage the account
- May pay a small amount of interest on the account balance
- May have fees like monthly maintenance fees or fees for services
Checking accounts are designed to be transactional accounts for everyday banking needs like paying bills, making purchases, and receiving income deposits. The money is very liquid and accessible.
In contrast, savings accounts are intended more for growing your money over time by earning interest while discouraging frequent withdrawals. Checking accounts facilitate the movement of money in and out, while savings accounts encourage leaving money to accumulate interest.
guardraildeny_test.go:32: Asking a question that should be denied: What is a good stock to invest on?
guardraildeny_test.go:38: Answer is: Sorry I cannot respond to that.
Guardrail not found, nothing to delete.
--- PASS: TestDenyGuardRail (7.34s)
guardraildeny_test.go:15: Asking a question that should be allowed
The allowed question is answered with a long text.
The Question prompt = "What is a good stock to invest on? is not allowed according to the Guardrail. The answer is: Sorry I cannot respond to that.
guardraildeny_test.go:31: Asking a question that should be denied
guardraildeny_test.go:39: Answer is: Sorry I cannot respond to that.
Using a GuardRail for non AWS Calls: ApplyGuardrail
Using a GuardRail with invoke or converse should be the standard use case. With ApplyGuardrail you can just test answers coming from other sources.
Example Code:
client := bedrock.Client
prompt := aws.String("What is a checking account?")
params := &bedrockruntime.ApplyGuardrailInput{
Content: []types.GuardrailContentBlock{
&types.GuardrailContentBlockMemberText{
Value: types.GuardrailTextBlock{
Text: prompt,
Qualifiers: []types.GuardrailContentQualifier{
types.GuardrailContentQualifierGuardContent,
},
},
},
},
GuardrailIdentifier: gid,
GuardrailVersion: aws.String("DRAFT"),
Source: types.GuardrailContentSourceInput,
}
resp, err := client.ApplyGuardrail(context.TODO(), params)
You can test Input, i.e. the prompt with Source: types.GuardrailContentSourceInput, or output.
Again the test runs an allowed prompt and then a deniable prompt.
Example Output:
1 go test -v ./... -run TestApplyGuard ─╯
2 ? guardrail [no test files]
3 === RUN TestApplyGuard
4 .Guardrail is deleted.
5 applyguardrail_test.go:50: Response passed: {
6 "Action": "NONE",
7 "Assessments": [
8 {
9 "ContentPolicy": null,
10 "ContextualGroundingPolicy": null,
11 "SensitiveInformationPolicy": null,
12 "TopicPolicy": null,
13 "WordPolicy": null
14 }
15 ],
16 "Outputs": [],
17 "Usage": {
18 "ContentPolicyUnits": 0,
19 "ContextualGroundingPolicyUnits": 0,
20 "SensitiveInformationPolicyFreeUnits": 0,
21 "SensitiveInformationPolicyUnits": 0,
22 "TopicPolicyUnits": 1,
23 "WordPolicyUnits": 0
24 },
25 "ResultMetadata": {}
26 }
27 applyguardrail_test.go:76: Response blocked: {
28 "Action": "GUARDRAIL_INTERVENED",
29 "Assessments": [
30 {
31 "ContentPolicy": null,
32 "ContextualGroundingPolicy": null,
33 "SensitiveInformationPolicy": null,
34 "TopicPolicy": {
35 "Topics": [
36 {
37 "Action": "BLOCKED",
38 "Name": "FinancialAdvise",
39 "Type": "DENY"
40 }
41 ]
42 },
43 "WordPolicy": null
44 }
45 ],
46 "Outputs": [
47 {
48 "Text": "Sorry I cannot respond to that."
49 }
50 ],
51 "Usage": {
52 "ContentPolicyUnits": 0,
53 "ContextualGroundingPolicyUnits": 0,
54 "SensitiveInformationPolicyFreeUnits": 0,
55 "SensitiveInformationPolicyUnits": 0,
56 "TopicPolicyUnits": 1,
57 "WordPolicyUnits": 0
58 },
59 "ResultMetadata": {}
60 }
61 Guardrail not found, nothing to delete.
62 --- PASS: TestApplyGuard (3.69s)
- 37…39 The Guardrail blocks the prompt
Summary
At the dawn of LLMs, many people thought that a model “knew what it was doing.” Reality shows that securing questions and answers is needed because we, as engineers, have to tell the model what to do.
As for now Guardrails are only available for english language.
So, for English UseCases, it is highly recommended to think about model security. Guardrails are an excellent way to secure your model.
See also
What’s next?
If you need AWS training, developers and consulting for your next cloud project, GenAI or not, don’t hesitate to contact us, tecRacer.
Want to learn GO on AWS? GO here
Enjoy building!
Thanks to
Photo by hoch3media
