Package net.stoerr.ai.aigenpipeline.framework.task

Class AIGenerationTask

java.lang.Object

net.stoerr.ai.aigenpipeline.framework.task.AIGenerationTask

All Implemented Interfaces:

Cloneable

public class AIGenerationTask
extends Object
implements Cloneable

We support the generation of files using an AI, specifically ChatGPT. A generation task can have several input files. Some of them can be prompt files with task descriptions, and some of them source files to be processed. The output of each task is one text file. A complex task can have several steps leading to several intermediate files.

Since ChatGPT is not particularily fast not free and the generation results have to be manually checked, this is heavily cached. Into each output file we write the versions of all the input files from which it was generated into a comment. When the tasks are run, we compare the versions of all the input files with the versions recorded in the comment, and only regenerate the output file if the versions have changed. An input file can have a version comment that explicitly states the version, or we take the an abbreviated SHA256 hash of the input file as version. It is possible to explicitly state the versions in version comments in the input files to avoid regenerating all files if minor details e.g. in a prompt file are changed - only when the prompt file version comment is changed everything is regenerated.

A version comment can e.g. look like this:

// AIGenVersion(ourversion, inputfile1@version1, inputfile2@version2, ...)

where ourversion and version1 and version2 are hashes. ourversion is the hash of the original output of the AI. The comment syntax (in this case //) is ignored - we just look for the AIGenVersion via regex.

Normally the intermediate and final results should be checked in with Git. That ensures manual checks when they are regenerated, and minimizes regeneration.

Field Summary

Fields

Modifier and Type	Field	Description
static String	FIXME	A marker that can be inserted by the AI when something is wrong / unclear.
protected List<File>	inputFiles
protected static Logger	LOG
protected Integer	maxTokens
protected File	outputFile
protected static Pattern	PATTERN_LICENCE	A pattern that matches the license header, which we want to remove to avoid clutter.
protected Map<String,String>	placeholdersAndValues
protected String	prompt	The actual prompt created from prompt files and parameters.
protected List<File>	promptFiles
protected RegenerationCheckStrategy	regenerationCheckStrategy
protected String	systemMessage
protected File	systemMessageFile
protected WritingStrategy	writingStrategy

Constructor Summary

Constructors

Constructor	Description
AIGenerationTask()

Method Summary

Modifier and Type	Method	Description
AIGenerationTask	addInputFile(File file)
AIGenerationTask	addInputFiles(List<File> files)
AIGenerationTask	addOptionalInputFile(File file)
AIGenerationTask	addPrompt(File promptFile, String... placeholdersAndValues)	The actual prompt to be executed.
AIGenerationTask	addPrompt(File promptFile, Map<String,String> placeholdersAndValues)	The actual prompt to be executed.
AIGenerationTask	copy()	Creates a deep copy of the task.
AIGenerationTask	execute(Supplier<AIChatBuilder> chatBuilderFactory, File rootDirectory)	Execute the task if necessary.
String	explain(Supplier<AIChatBuilder> chatBuilderFactory, File rootDirectory, String question)	Ask a question about the previous task execution.
protected List<String>	getAdditionalMarkers()
protected List<File>	getAllInputFiles()
protected String	getFileContent(File file)
boolean	hasToBeRun()
protected AIChatBuilder	makeChatBuilder(Supplier<AIChatBuilder> chatBuilderFactory, File rootDirectory, String outputRelPath)
AIGenerationTask	maxTokens(Integer maxTokens)
protected String	relativePath(File file, File rootDirectory)
AIGenerationTask	setOutputFile(File file)
AIGenerationTask	setRegenerationCheckStrategy(RegenerationCheckStrategy strategy)	Sets the strategy to check whether the generation has to be run.
AIGenerationTask	setSystemMessage(File systemMessageFile)
AIGenerationTask	setWritingStrategy(WritingStrategy strategy)	Sets the strategy to deal with writing the output file.
String	toJson(Supplier<AIChatBuilder> chatBuilderFactory, File rootDirectory)	For debugging purposes: returns the JSON that would be sent to the AI.
String	toString()
protected static String	unclutter(String content)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

protected static final Logger LOG

FIXME

public static final String FIXME

A marker that can be inserted by the AI when something is wrong / unclear. We will make sure the user sees that by aborting.

See Also:

Constant Field Values

PATTERN_LICENCE

protected static final Pattern PATTERN_LICENCE

A pattern that matches the license header, which we want to remove to avoid clutter.

inputFiles

protected List<File> inputFiles

outputFile

protected File outputFile

prompt

protected String prompt

The actual prompt created from prompt files and parameters.

promptFiles

protected List<File> promptFiles

placeholdersAndValues

protected Map<String,String> placeholdersAndValues

systemMessage

protected String systemMessage

systemMessageFile

protected File systemMessageFile

maxTokens

protected Integer maxTokens

regenerationCheckStrategy

protected RegenerationCheckStrategy regenerationCheckStrategy

writingStrategy

protected WritingStrategy writingStrategy

Constructor Detail

AIGenerationTask

public AIGenerationTask()

Method Detail

copy

public AIGenerationTask copy()

Creates a deep copy of the task.

maxTokens

public AIGenerationTask maxTokens(Integer maxTokens)

addOptionalInputFile

public AIGenerationTask addOptionalInputFile(@Nullable
                                             File file)

addInputFiles

public AIGenerationTask addInputFiles(List<File> files)

addInputFile

public AIGenerationTask addInputFile(File file)

setOutputFile

public AIGenerationTask setOutputFile(@Nonnull
                                      File file)

setRegenerationCheckStrategy

public AIGenerationTask setRegenerationCheckStrategy(RegenerationCheckStrategy strategy)

Sets the strategy to check whether the generation has to be run. Default is RegenerationCheckStrategy.VERSIONMARKER.

setWritingStrategy

public AIGenerationTask setWritingStrategy(WritingStrategy strategy)

Sets the strategy to deal with writing the output file. Default is WritingStrategy.WITHVERSION.

hasToBeRun

public boolean hasToBeRun()
                   throws IOException

Throws:

IOException

getAdditionalMarkers

protected List<String> getAdditionalMarkers()

getAllInputFiles

protected List<File> getAllInputFiles()

addPrompt

public AIGenerationTask addPrompt(@Nonnull
                                  File promptFile,
                                  String... placeholdersAndValues)

The actual prompt to be executed. The prompt file content can contain placeholders that are replaced by the values given: placeholdersAndValues contain alternatingly placeholder names and values for them.

Returns:

this

addPrompt

public AIGenerationTask addPrompt(@Nonnull
                                  File promptFile,
                                  Map<String,String> placeholdersAndValues)

The actual prompt to be executed. The prompt file content can contain placeholders that are replaced by the values given.

Returns:

this

getFileContent

protected String getFileContent(@Nonnull
                                File file)

unclutter

protected static String unclutter(String content)

setSystemMessage

public AIGenerationTask setSystemMessage(@Nonnull
                                         File systemMessageFile)

relativePath

protected String relativePath(@Nullable
                              File file,
                              @Nonnull
                              File rootDirectory)

execute

public AIGenerationTask execute(@Nonnull
                                Supplier<AIChatBuilder> chatBuilderFactory,
                                @Nonnull
                                File rootDirectory)
                         throws IOException

Execute the task if necessary. If the output file is already there and up to date, nothing is done.

Throws:

IOException

toJson

public String toJson(@Nonnull
                     Supplier<AIChatBuilder> chatBuilderFactory,
                     @Nonnull
                     File rootDirectory)

For debugging purposes: returns the JSON that would be sent to the AI.

makeChatBuilder

@Nonnull
protected AIChatBuilder makeChatBuilder(@Nonnull
                                        Supplier<AIChatBuilder> chatBuilderFactory,
                                        @Nonnull
                                        File rootDirectory,
                                        String outputRelPath)

explain

public String explain(@Nonnull
                      Supplier<AIChatBuilder> chatBuilderFactory,
                      @Nonnull
                      File rootDirectory,
                      @Nonnull
                      String question)
               throws IOException

Ask a question about the previous task execution. We assume it was previously run (hasToBeRun() == false), add the result of the previous execution to the chat, and ask the AI the given question about it. This can be used e.g. to see why the AI did something, or in the process of improving the prompt, etc.

Returns:

the answer of the AI - not written to a file!

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object