🏓 Survey Papers

Abstract

This chapter presents brief overivew of some of the popular survey papers related to Generative AI and Prompt Engineering.

[1] A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Overview: The paper discusses the significance of GPT-3 family LLMs like GPT-3, ChatGPT, and GPT4, which have gained popularity due to their remarkable natural language processing abilities achieved through large-scale pretraining.

The survey covers foundational concepts from transformers to large language models including transfer learning and self-supervised learning, provides an overview of GLLMs, discusses their performance in different tasks and languages, presents their data-related capabilities, robustness, and effectiveness as evaluators, and concludes by suggesting future research directions.

Overall, this survey is a valuable resource for academia and industry researchers to stay updated on GPT-3 family LLMs.

[2] A Survey of Large Language Models

Overview: The paper provides an overview of the development and progression of large language models (LLMs), emphasizing their remarkable proficiency in addressing a wide range of natural language processing (NLP) challenges.

This paper highlights the significance of model scaling, where increasing the model size leads to significant performance improvements and unique abilities not present in smaller models.

This survey comprehensively explores and discusses pivotal facets of LLMs, including pre-training, adaptation tuning, utilization, and capacity evaluation, while also presenting existing resources and future research directions in the field.

[3] Challenges and Applications of Large Language Models

Overview: The paper outlines the swift advancement of Large Language Models (LLMs) within the realm of artificial intelligence and underscores the challenges in keeping up with their progress.

To tackle this issue, the authors propose a systematic framework to identify open problems and successful applications of LLMs, helping AI researchers gain a better understanding of the current state of the field and become more productive.

[4] A Comprehensive Survey of AI-Generated Content (AIGC):A History of Generative AI from GAN to ChatGPT

Overview: The paper explores the increasing attention directed towards ChatGPT, DALL-E-2, and Codex, with a focus on their contributions to the field of Artificial Intelligence Generated Content (AIGC).

The paper offers a comprehensive survey of generative models' history, components, recent advances in unimodal and multimodal interaction, and highlights open problems and forthcoming challenges in the realm of AIGC.

[5] A Survey on ChatGPT: AI–Generated Contents, Challenges, and Solutions

Overview: The paper delves into the increasing importance of AI-generated content (AIGC), driven by large AI models like ChatGPT, in transforming the landscape of content generation and knowledge representation. It emphasizes the potential of AIGC to produce high-quality content rapidly and cost-effectively based on user prompts but underscores the need to address security, privacy, ethical, and legal challenges.

The paper provides an in-depth survey of AIGC's working principles, security threats, privacy concerns, and presents watermarking approaches while highlighting prospective avenues for further research in this domain.

[6] Prompting Frameworks for Large Language Models: A Survey

Overview: The paper provides an in-depth exploration of the rapid advancements in large language models (LLMs) like ChatGPT and their impact across diverse domains. It underscores the inherent limitations of LLMs, such as temporal lag in training data and the inability to perform physical actions.

The paper introduces the novel concept of a "Prompting Framework" (PF) to manage and simplify interactions with LLMs, outlining its hierarchical structure and prospective avenues for future research areas in this emerging field.

[7] ChatGPT’s One-year Anniversary: Are Open-Source Large Language Models Catching up?

Overview: In the latter part of 2022, ChatGPT made a significant impact on the AI landscape, showcasing the potential of large language models (LLMs) through supervised fine-tuning and reinforcement learning from human feedback.

Although proprietary LLMs such as ChatGPT and Claude generally exhibited superior performance compared to their open-source counterparts, the open-source LLMs progressed rapidly and, in some cases, demonstrated comparable or even superior performance on specific tasks.

This paper, on the first anniversary of ChatGPT, offers an extensive survey of open-source LLMs that have achieved performance levels equivalent to or surpassing ChatGPT across a range of tasks.

[8] The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Overview: The paper presents the rapid growth of Large Language Models (LLMs) and the challenges posed by their increasing computational demands.

It presents a comprehensive review of algorithmic advancements aimed at enhancing the efficiency of LLMs, covering various topics such as scaling laws, data utilization, architectural innovations, training strategies, and inference techniques.

The paper aims to be a valuable resource for researchers and practitioners in the field, facilitating future advancements in the development of LLMs.

[9] A Survey on In-context Learning

Overview: This paper explores the emerging paradigm of in-context learning (ICL) in natural language processing, driven by the capabilities of large language models (LLMs).

The authors provide a precise definition of ICL, discuss its relationship with related studies, and delve into advanced techniques such as training and demonstration strategies.

They also highlight the challenges of ICL and suggest potential research directions, with the overarching goal of stimulating further investigation and improvement in this area.

[10] Retrieval-Augmented Generation for Large Language Models: A Survey

Overview: The paper discusses Retrieval-Augmented Generation (RAG), an important technique aimed at enhancing the performance of large language models (LLMs) by incorporating external knowledge bases.

RAG improves answer accuracy, reduces model hallucination, and increases transparency by allowing users to verify answers using source citations.

The paper outlines the development paradigms of RAG, its core components (retriever, generator, and augmentation methods), evaluation methods, and future research directions, with a special focus on vertical optimization, horizontal scalability, and the technical stack and ecosystem of RAG in the context of LLMs.