Initial Query

Why isn't this working

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@app.route('/getsimilar', methods=['GET'])
def getsimilar():
    print('args',request.args)
    positive = request.args.get('positive', '')
    positive = positive.split("##")

    negative = request.args.get('negative', '')
    negative = negative.split("##")

    keep_yes = [i for i,x in enumerate(titles) if x in positive]
    set1_yes = emb[keep_yes]

    keep_no = [i for i,x in enumerate(titles) if x in negative]
    set1_no = emb[keep_no]

    print("OK", len(set1_yes), len(set1_no))

1
2
3

args ImmutableMultiDict([('positive', 'Title: Towards Evaluating the Robustness of Neural Networks
Authors: Nicholas Carlini; David Wagner'), ('negative', 'Title: On Evaluating Adversarial Robustness Authors: Nicholas Carlini; Anish Athalye; Nicolas Papernot; Wieland Brendel; Jonas Rauber; Dimitris Tsipras; Ian Goodfellow; Aleksander Madry; Alexey Kurakin}')])
OK 1 0

LLM Generation

Query

Why isn't this working

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@app.route('/getsimilar', methods=['GET'])
def getsimilar():
    print('args',request.args)
    positive = request.args.get('positive', '')
    positive = positive.split("##")

    negative = request.args.get('negative', '')
    negative = negative.split("##")

    keep_yes = [i for i,x in enumerate(titles) if x in positive]
    set1_yes = emb[keep_yes]

    keep_no = [i for i,x in enumerate(titles) if x in negative]
    set1_no = emb[keep_no]

    print("OK", len(set1_yes), len(set1_no))

1
2
3

args ImmutableMultiDict([('positive', 'Title: Towards Evaluating the Robustness of Neural Networks
Authors: Nicholas Carlini; David Wagner'), ('negative', 'Title: On Evaluating Adversarial Robustness Authors: Nicholas Carlini; Anish Athalye; Nicolas Papernot; Wieland Brendel; Jonas Rauber; Dimitris Tsipras; Ian Goodfellow; Aleksander Madry; Alexey Kurakin}')])
OK 1 0

Output

**Reason:** The string value provided for the `negative` parameter (`'Title: On Evaluating Adversarial Robustness Authors: ...'`) does not exactly match any string element within your `titles` list. The list comprehension `if x in negative` requires an exact match, and subtle differences like missing newlines (`\n`), extra spaces, or other formatting variations between the input string and the string stored in `titles` will prevent a match. **Change:** Ensure the string sent in the `negative` query parameter *exactly* matches the corresponding string stored in the `titles` list, including any newlines or specific spacing. For example, if the entry in `titles` includes a newline like the positive example, you would need to send the negative title similarly formatted in the URL (URL-encoded).

LLM Generation

Query

I asked a question about why some code was incorrect. The answer is "because the input contains an extra curly brace }". Does this student answer say that? **Reason:** The string value provided for the `negative` parameter (`'Title: On Evaluating Adversarial Robustness Authors: ...'`) does not exactly match any string element within your `titles` list. The list comprehension `if x in negative` requires an exact match, and subtle differences like missing newlines (`\n`), extra spaces, or other formatting variations between the input string and the string stored in `titles` will prevent a match. **Change:** Ensure the string sent in the `negative` query parameter *exactly* matches the corresponding string stored in the `titles` list, including any newlines or specific spacing. For example, if the entry in `titles` includes a newline like the positive example, you would need to send the negative title similarly formatted in the URL (URL-encoded). Answer either 'The student passes' or 'The student fails'.

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False