Initial Query

Why isn't this working 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
@app.route('/getsimilar', methods=['GET'])
def getsimilar():
    print('args',request.args)
    positive = request.args.get('positive', '')
    positive = positive.split("##")

    negative = request.args.get('negative', '')
    negative = negative.split("##")

    keep_yes = [i for i,x in enumerate(titles) if x in positive]
    set1_yes = emb[keep_yes]

    keep_no = [i for i,x in enumerate(titles) if x in negative]
    set1_no = emb[keep_no]

    print("OK", len(set1_yes), len(set1_no))
I see 
1
2
3
args ImmutableMultiDict([('positive', 'Title: Towards Evaluating the Robustness of Neural Networks
Authors: Nicholas Carlini; David Wagner'), ('negative', 'Title: On Evaluating Adversarial Robustness Authors: Nicholas Carlini; Anish Athalye; Nicolas Papernot; Wieland Brendel; Jonas Rauber; Dimitris Tsipras; Ian Goodfellow; Aleksander Madry; Alexey Kurakin}')])
OK 1 0
Why don't I see OK 1 1? Instead it just says OK 1 0. List exactly one reason explaining what's wrong and what one change would fix it.

LLM Generation

Query

Why isn't this working 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
@app.route('/getsimilar', methods=['GET'])
def getsimilar():
    print('args',request.args)
    positive = request.args.get('positive', '')
    positive = positive.split("##")

    negative = request.args.get('negative', '')
    negative = negative.split("##")

    keep_yes = [i for i,x in enumerate(titles) if x in positive]
    set1_yes = emb[keep_yes]

    keep_no = [i for i,x in enumerate(titles) if x in negative]
    set1_no = emb[keep_no]

    print("OK", len(set1_yes), len(set1_no))
I see 
1
2
3
args ImmutableMultiDict([('positive', 'Title: Towards Evaluating the Robustness of Neural Networks
Authors: Nicholas Carlini; David Wagner'), ('negative', 'Title: On Evaluating Adversarial Robustness Authors: Nicholas Carlini; Anish Athalye; Nicolas Papernot; Wieland Brendel; Jonas Rauber; Dimitris Tsipras; Ian Goodfellow; Aleksander Madry; Alexey Kurakin}')])
OK 1 0
Why don't I see OK 1 1? Instead it just says OK 1 0. List exactly one reason explaining what's wrong and what one change would fix it.

Output

The issue is that the `negative` argument in the request has an unmatched closing curly brace `}` at the end of its title ("Alexey Kurakin}"). This causes `x in negative` to fail because the title in `titles` likely doesn't include the `}`. **Fix**: Remove the stray `}` from the negative argument's title. Change: 

1
... 'Alexey Kurakin}'
to 
1
... 'Alexey Kurakin'
so it matches the title in your data.

LLM Generation

Query

I asked a question about why some code was incorrect. The answer is "because the input contains an extra curly brace }". Does this student answer say that? The issue is that the `negative` argument in the request has an unmatched closing curly brace `}` at the end of its title ("Alexey Kurakin}"). This causes `x in negative` to fail because the title in `titles` likely doesn't include the `}`. **Fix**: Remove the stray `}` from the negative argument's title. Change: 

1
... 'Alexey Kurakin}'
to 
1
... 'Alexey Kurakin'
so it matches the title in your data. Answer either 'The student passes' or 'The student fails'.

Output

The student passes.

Substring Evaluation

Testing if the previous output contains the string student passes: True