The main workflow is made up of two nested sub-workflows:
summary_gen: This sub-workflow finds research papers on the given topic and generates summaries. It carries out the searching for papers by web querying and uses LLM to get insights and summaries as instructed.slide_gen: this sub-workflow is responsible for generating a PowerPoint slide deck using the summaries from the previous step. It formats the slides using a provided PowerPoint template and generates them by creating and executing Python code using thepython-pptxlibrary.
Let’s take a closer look at the these sub-workflows. Firstly, the summary_gen workflow, which is pretty straightforward. It follows a simple linear process. It basically serves as a “data processing” workflow, with some steps sending a request to an LLM.
The workflow starts by getting a user input (a research topic) and run through the following steps:
tavily_query: Queries with the Tavily API to get academic papers related to the topic as a structured response.get_paper_with_citations: For each paper returned from the Tavily query, the step retrieves the paper metadata along with that of the cited paper using the SemanticScholar API.filter_papers: Since not all citations retrieved are directly relevant to the original topic, this step refines the result. The titles and abstracts of each paper are sent to the LLM to assess their relevance. This step is defined as:
@step(num_workers=4)
async def filter_papers(self, ev: PaperEvent) -> FilteredPaperEvent:
llm = new_gpt4o_mini(temperature=0.0)
response = await process_citation(ev.paper, llm)
return FilteredPaperEvent(paper=ev.paper, is_relevant=response)
Here in the process_citation() function, we use the FunctionCallingProgram from LlamaIndex to get a structured response:
IS_CITATION_RELEVANT_PMT = """
You help a researcher decide whether a paper is relevant to their current research topic: {topic}
You are given the title and abstract of a paper.
title: {title}
abstract: {abstract}Give a score indicating the relevancy to the research topic, where:
Score 0: Not relevant
Score 1: Somewhat relevant
Score 2: Very relevant
Answer with integer score 0, 1 or 2 and your reason.
"""
class IsCitationRelevant(BaseModel):
score: int
reason: str
async def process_citation(citation, llm):
program = FunctionCallingProgram.from_defaults(
llm=llm,
output_cls=IsCitationRelevant,
prompt_template_str=IS_CITATION_RELEVANT_PMT,
verbose=True,
)
response = await program.acall(
title=citation.title,
abstract=citation.summary,
topic=citation.topic,
description="Data model for whether the paper is relevant to the research topic.",
)
return response
download_papers: This step gathers all filtered papers, prioritizes them based on relevance score and availability on ArXiv, and downloads the most relevant ones.paper2summary_dispatcher: Each downloaded paper is prepared for summary generation by setting up paths for storing the images and the summaries. This step usesself.send_event()to enable the parallel execution of thepaper2summarystep for each paper. It also sets the number of papers in the workflow context with a variablectx.data[“n_pdfs”]so that the later steps know how many papers they are expected to process in total.
@step(pass_context=True)
async def paper2summary_dispatcher(
self, ctx: Context, ev: Paper2SummaryDispatcherEvent
) -> Paper2SummaryEvent:
ctx.data["n_pdfs"] = 0
for pdf_name in Path(ev.papers_path).glob("*.pdf"):
img_output_dir = self.papers_images_path / pdf_name.stem
img_output_dir.mkdir(exist_ok=True, parents=True)
summary_fpath = self.paper_summary_path / f"{pdf_name.stem}.md"
ctx.data["n_pdfs"] += 1
self.send_event(
Paper2SummaryEvent(
pdf_path=pdf_name,
image_output_dir=img_output_dir,
summary_path=summary_fpath,
)
)
paper2summary: For each paper, it converts the PDF into images, which are then sent to the LLM for summarization. Once the summary is generated, it is saved in a markdown file for future reference. Particularly, the summary generated here is quite elaborated, like a small article, so not quite suitable yet for putting directly in the presentation. But it is kept so that the user can view these intermediate results. In one of the later steps, we will make this information more presentable. The prompt provided to the LLM includes key instructions to ensure accurate and concise summaries:
SUMMARIZE_PAPER_PMT = """
You are an AI specialized in summarizing scientific papers.
Your goal is to create concise and informative summaries, with each section preferably around 100 words and
limited to a maximum of 200 words, focusing on the core approach, methodology, datasets,
evaluation details, and conclusions presented in the paper. After you summarize the paper,
save the summary as a markdown file.Instructions:
- Key Approach: Summarize the main approach or model proposed by the authors.
Focus on the core idea behind their method, including any novel techniques, algorithms, or frameworks introduced.
- Key Components/Steps: Identify and describe the key components or steps in the model or approach.
Break down the architecture, modules, or stages involved, and explain how each contributes to the overall method.
- Model Training/Finetuning: Explain how the authors trained or finetuned their model.
Include details on the training process, loss functions, optimization techniques,
and any specific strategies used to improve the model’s performance.
- Dataset Details: Provide an overview of the datasets used in the study.
Include information on the size, type and source. Mention whether the dataset is publicly available
and if there are any benchmarks associated with it.
- Evaluation Methods and Metrics: Detail the evaluation process used to assess the model's performance.
Include the methods, benchmarks, and metrics employed.
- Conclusion: Summarize the conclusions drawn by the authors. Include the significance of the findings,
any potential applications, limitations acknowledged by the authors, and suggested future work.
Ensure that the summary is clear and concise, avoiding unnecessary jargon or overly technical language.
Aim to be understandable to someone with a general background in the field.
Ensure that all details are accurate and faithfully represent the content of the original paper.
Avoid introducing any bias or interpretation beyond what is presented by the authors. Do not add any
information that is not explicitly stated in the paper. Stick to the content presented by the authors.
"""
finish: The workflow collects all generated summaries, verifies they are correctly stored, and logs the completion of the process, and return aStopEventas a final result.
If this workflow were to run independently, execution would end here. However, since this is just a sub-workflow of the main process, upon completion, the next sub-workflow — slide_gen — is triggered.
This workflow generates slides based on the summaries created in the previous step. Here is an overview of the slide_gen workflow:
When the previous sub-workflow finishes, and the summary markdown files are ready, this workflow starts:
get_summaries: This step reads the content of the summary files, triggers aSummaryEventfor each file utilizing againself.send_event()to enable concurrent execution for faster processing.summary2outline: This step makes summaries into slide outline texts by using LLM. It shortens the summaries into sentences or bullet points for putting in the presentation.gather_feedback_outline: In this step, it presents the the user the proposed slide outline alongside the paper summary for them to review. The user provides feedback, which may trigger anOutlineFeedbackEventif revisions are necessary. This feedback loop continues with thesummary2outlinestep until the user approves the final outline, at which point anOutlineOkEventis triggered.
@step(pass_context=True)
async def gather_feedback_outline(
self, ctx: Context, ev: OutlineEvent
) -> OutlineFeedbackEvent | OutlineOkEvent:
"""Present user the original paper summary and the outlines generated, gather feedback from user"""
print(f"the original summary is: {ev.summary}")
print(f"the outline is: {ev.outline}")
print("Do you want to proceed with this outline? (yes/no):")
feedback = input()
if feedback.lower().strip() in ["yes", "y"]:
return OutlineOkEvent(summary=ev.summary, outline=ev.outline)
else:
print("Please provide feedback on the outline:")
feedback = input()
return OutlineFeedbackEvent(
summary=ev.summary, outline=ev.outline, feedback=feedback
)
outlines_with_layout: It augments every slide outline by including page layout details from the given PowerPoint template, using LLM. This stage saves the content and design for all slide pages in a JSON file.slide_gen: It uses a ReAct agent to make slide decks based on given outlines and layout details. This agent has a code interpreter tool to run and correct code in an isolated environment and a layout-checking tool to look at the given PowerPoint template information. The agent is prompted to usepython-pptxto create the slides and can observe and fix mistakes.
@step(pass_context=True)
async def slide_gen(
self, ctx: Context, ev: OutlinesWithLayoutEvent
) -> SlideGeneratedEvent:
agent = ReActAgent.from_tools(
tools=self.azure_code_interpreter.to_tool_list() + [self.all_layout_tool],
llm=new_gpt4o(0.1),
verbose=True,
max_iterations=50,
)prompt = (
SLIDE_GEN_PMT.format(
json_file_path=ev.outlines_fpath.as_posix(),
template_fpath=self.slide_template_path,
final_slide_fname=self.final_slide_fname,
)
+ REACT_PROMPT_SUFFIX
)
agent.update_prompts({"agent_worker:system_prompt": PromptTemplate(prompt)})
res = self.azure_code_interpreter.upload_file(
local_file_path=self.slide_template_path
)
logging.info(f"Uploaded file to Azure: {res}")
response = agent.chat(
f"An example of outline item in json is {ev.outline_example.json()},"
f" generate a slide deck"
)
local_files = self.download_all_files_from_session()
return SlideGeneratedEvent(
pptx_fpath=f"{self.workflow_artifacts_path}/{self.final_slide_fname}"
)
validate_slides: Checks the slide deck to make sure it meets the given standards. This step involves turning the slides into images and having the LLM visually inspect them for correct content and consistent style according to the guidelines. Depending on what the LLM finds, it will either send out aSlideValidationEventif there are problems or aStopEventif everything looks good.
@step(pass_context=True)
async def validate_slides(
self, ctx: Context, ev: SlideGeneratedEvent
) -> StopEvent | SlideValidationEvent:
"""Validate the generated slide deck"""
ctx.data["n_retry"] += 1
ctx.data["latest_pptx_file"] = Path(ev.pptx_fpath).name
img_dir = pptx2images(Path(ev.pptx_fpath))
image_documents = SimpleDirectoryReader(img_dir).load_data()
llm = mm_gpt4o
program = MultiModalLLMCompletionProgram.from_defaults(
output_parser=PydanticOutputParser(SlideValidationResult),
image_documents=image_documents,
prompt_template_str=SLIDE_VALIDATION_PMT,
multi_modal_llm=llm,
verbose=True,
)
response = program()
if response.is_valid:
return StopEvent(
self.workflow_artifacts_path.joinpath(self.final_slide_fname)
)
else:
if ctx.data["n_retry"] < self.max_validation_retries:
return SlideValidationEvent(result=response)
else:
return StopEvent(
f"The slides are not fixed after {self.max_validation_retries} retries!"
)
The criteria used for validation are:
SLIDE_VALIDATION_PMT = """
You are an AI that validates the slide deck generated according to following rules:
- The slide need to have a front page
- The slide need to have a final page (e.g. a 'thank you' or 'questions' page)
- The slide texts are clearly readable, not cut off, not overflowing the textbox
and not overlapping with other elementsIf any of the above rules are violated, you need to provide the index of the slide that violates the rule,
as well as suggestion on how to fix it.
"""
modify_slides: Should the slides fail the validation check, the previous step sends aSlideValidationEventevent. Here another ReAct agent updates the slides according to validator feedback, with the updated slides being saved and returned to be validated again. This verification loop could occur several times according to themax_validation_retriesvariable attributes of theSlideGenWorkflowclass.