Initial Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

LLM Generation

Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

Output

{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "Spatial Transformer Networks", "author": "Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu", "year": "2015"}, 3: {"title": "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", "author": "Sergey Ioffe, Christian Szegedy", "year": "2015"}, 4: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 5: {"title": "Rethinking the Inception Architecture for Computer Vision", "author": "Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, Zbigniew Wojna", "year": "2016"}, 6: {"title": "Densely Connected Convolutional Networks", "author": "Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger", "year": "2017"} }

LLM Generation

Query

I asked the student for the references contained a paper that didn't have any references. The student answered "{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "Spatial Transformer Networks", "author": "Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu", "year": "2015"}, 3: {"title": "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", "author": "Sergey Ioffe, Christian Szegedy", "year": "2015"}, 4: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 5: {"title": "Rethinking the Inception Architecture for Computer Vision", "author": "Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, Zbigniew Wojna", "year": "2016"}, 6: {"title": "Densely Connected Convolutional Networks", "author": "Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger", "year": "2017"} }" Does the student's answer say something like "There are no references"? If so, the student passes, and write "The student passes". Otherwise, if the student lists *any* paper, write "The student fails".

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False

Initial Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

LLM Generation

Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

Output

{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "YOLO9000: Better, Faster, Stronger", "author": "Joseph Redmon, Ali Farhadi", "year": "2016"}, 3: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 4: {"title": "U-Net: Convolutional Networks for Biomedical Image Segmentation", "author": "Olaf Ronneberger, Philipp Fischer, Thomas Brox", "year": "2015"}, 5: {"title": "Attention is All You Need", "author": "Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin", "year": "2017"}, 6: {"title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", "author": "Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova", "year": "2018"} }

LLM Generation

Query

I asked the student for the references contained a paper that didn't have any references. The student answered "{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "YOLO9000: Better, Faster, Stronger", "author": "Joseph Redmon, Ali Farhadi", "year": "2016"}, 3: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 4: {"title": "U-Net: Convolutional Networks for Biomedical Image Segmentation", "author": "Olaf Ronneberger, Philipp Fischer, Thomas Brox", "year": "2015"}, 5: {"title": "Attention is All You Need", "author": "Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin", "year": "2017"}, 6: {"title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", "author": "Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova", "year": "2018"} }" Does the student's answer say something like "There are no references"? If so, the student passes, and write "The student passes". Otherwise, if the student lists *any* paper, write "The student fails".

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False

Initial Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

LLM Generation

Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

Output

{ 1: {"title": "A Survey of Techniques for Event Detection in Twitter", "author": "Gupta, A., & Kumaraguru, P.", "year": "2012"}, 2: {"title": "Twitter as a Rapid Response News Service: An Exploration in the Context of the 2008 Mumbai Blasts", "author": "Vieweg, S., Hughes, A. L., Starbird, K., & Palen, L.", "year": "2010"}, 3: {"title": "TwitterStand: News in Tweets", "author": "Sankaranarayanan, J., Samet, H., Teitler, B., & Lieberman, M. D.", "year": "2009"}, 4: {"title": "Adaptive Real-time Event Detection Using Social Media Streams", "author": "Sakaki, T., Okazaki, M., & Matsuo, Y.", "year": "2010"}, 5: {"title": "Event Detection in Twitter", "author": "Krishnamurthy, B., Gill, P., & Arlitt, M.", "year": "2013"}, 6: {"title": "TwitterMonitor: Trend Detection over the Twitter Stream", "author": "Doan, S., Ma, D., & Halevy, A.", "year": "2011"} }

LLM Generation

Query

I asked the student for the references contained a paper that didn't have any references. The student answered "{ 1: {"title": "A Survey of Techniques for Event Detection in Twitter", "author": "Gupta, A., & Kumaraguru, P.", "year": "2012"}, 2: {"title": "Twitter as a Rapid Response News Service: An Exploration in the Context of the 2008 Mumbai Blasts", "author": "Vieweg, S., Hughes, A. L., Starbird, K., & Palen, L.", "year": "2010"}, 3: {"title": "TwitterStand: News in Tweets", "author": "Sankaranarayanan, J., Samet, H., Teitler, B., & Lieberman, M. D.", "year": "2009"}, 4: {"title": "Adaptive Real-time Event Detection Using Social Media Streams", "author": "Sakaki, T., Okazaki, M., & Matsuo, Y.", "year": "2010"}, 5: {"title": "Event Detection in Twitter", "author": "Krishnamurthy, B., Gill, P., & Arlitt, M.", "year": "2013"}, 6: {"title": "TwitterMonitor: Trend Detection over the Twitter Stream", "author": "Doan, S., Ma, D., & Halevy, A.", "year": "2011"} }" Does the student's answer say something like "There are no references"? If so, the student passes, and write "The student passes". Otherwise, if the student lists *any* paper, write "The student fails".

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False

Initial Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

LLM Generation

Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

Output

{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "Mask R-CNN", "author": "Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick", "year": "2017"}, 3: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 4: {"title": "YOLO9000: Better, Faster, Stronger", "author": "Joseph Redmon, Ali Farhadi", "year": "2016"}, 5: {"title": "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2015"}, 6: {"title": "Densely Connected Convolutional Networks", "author": "Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger", "year": "2017"} }

LLM Generation

Query

I asked the student for the references contained a paper that didn't have any references. The student answered "{ 1: {"title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016"}, 2: {"title": "Mask R-CNN", "author": "Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick", "year": "2017"}, 3: {"title": "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", "author": "Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun", "year": "2015"}, 4: {"title": "YOLO9000: Better, Faster, Stronger", "author": "Joseph Redmon, Ali Farhadi", "year": "2016"}, 5: {"title": "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2015"}, 6: {"title": "Densely Connected Convolutional Networks", "author": "Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger", "year": "2017"} }" Does the student's answer say something like "There are no references"? If so, the student passes, and write "The student passes". Otherwise, if the student lists *any* paper, write "The student fails".

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False

Initial Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

LLM Generation

Query

For each of the 6 reference in the following paper, extract the title, author, and year as a JSON array. Write {1: {"title": "title", "author": "author", "year": "year"}, 1: ..., 6: ...} Paper: {paper_title} {paper_body} {references}

Output

{ 1: { "title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016" }, 2: { "title": "Identity Mappings in Deep Residual Networks", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016" }, 3: { "title": "Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models", "author": "Sergey Ioffe, Christian Szegedy", "year": "2017" }, 4: { "title": "Residual Networks Behave Like Ensembles of Relatively Shallow Networks", "author": "Andreas Veit, Michael Wilber, Serge Belongie", "year": "2016" }, 5: { "title": "Wide Residual Networks", "author": "Sergey Zagoruyko, Nikos Komodakis", "year": "2016" }, 6: { "title": "Rethinking the Inception Architecture for Computer Vision", "author": "Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna", "year": "2016" } }

LLM Generation

Query

I asked the student for the references contained a paper that didn't have any references. The student answered "{ 1: { "title": "Deep Residual Learning for Image Recognition", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016" }, 2: { "title": "Identity Mappings in Deep Residual Networks", "author": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun", "year": "2016" }, 3: { "title": "Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models", "author": "Sergey Ioffe, Christian Szegedy", "year": "2017" }, 4: { "title": "Residual Networks Behave Like Ensembles of Relatively Shallow Networks", "author": "Andreas Veit, Michael Wilber, Serge Belongie", "year": "2016" }, 5: { "title": "Wide Residual Networks", "author": "Sergey Zagoruyko, Nikos Komodakis", "year": "2016" }, 6: { "title": "Rethinking the Inception Architecture for Computer Vision", "author": "Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna", "year": "2016" } }" Does the student's answer say something like "There are no references"? If so, the student passes, and write "The student passes". Otherwise, if the student lists *any* paper, write "The student fails".

Output

The student fails.

Substring Evaluation

Testing if the previous output contains the string student passes: False