Package net.stoerr.ai.aigenpipeline.framework.task

Class AIGenerationTask

java.lang.Object
net.stoerr.ai.aigenpipeline.framework.task.AIGenerationTask
All Implemented Interfaces:
Cloneable
public class AIGenerationTask extends Object implements Cloneable
We support the generation of files using an AI, specifically ChatGPT. A generation task can have several input files. Some of them can be prompt files with task descriptions, and some of them source files to be processed. The output of each task is one text file. A complex task can have several steps leading to several intermediate files.

Since ChatGPT is not particularily fast not free and the generation results have to be manually checked, this is heavily cached. Into each output file we write the versions of all the input files from which it was generated into a comment. When the tasks are run, we compare the versions of all the input files with the versions recorded in the comment, and only regenerate the output file if the versions have changed. An input file can have a version comment that explicitly states the version, or we take the an abbreviated SHA256 hash of the input file as version. It is possible to explicitly state the versions in version comments in the input files to avoid regenerating all files if minor details e.g. in a prompt file are changed - only when the prompt file version comment is changed everything is regenerated.

A version comment can e.g. look like this:

// AIGenVersion(ourversion, inputfile1@version1, inputfile2@version2, ...)

where ourversion and version1 and version2 are hashes. ourversion is the hash of the original output of the AI. The comment syntax (in this case //) is ignored - we just look for the AIGenVersion via regex.

Normally the intermediate and final results should be checked in with Git. That ensures manual checks when they are regenerated, and minimizes regeneration.

  • Field Details

    • LOG

      protected static final Logger LOG

    • FIXME

      public static final String FIXME
      A marker that can be inserted by the AI when something is wrong / unclear. We will make sure the user sees that by aborting.
      See Also:

    • PATTERN_LICENCE

      protected static final Pattern PATTERN_LICENCE
      A pattern that matches the license header, which we want to remove to avoid clutter.

    • inputFiles

      protected List<File> inputFiles

    • outputFile

      protected File outputFile

    • prompt

      protected String prompt
      The actual prompt created from prompt files and parameters.

    • promptFiles

      protected List<File> promptFiles

    • placeholdersAndValues

      protected Map<String,String> placeholdersAndValues

    • systemMessage

      protected String systemMessage

    • systemMessageFile

      protected File systemMessageFile

    • force

      protected boolean force

    • maxTokens

      protected Integer maxTokens

  • Constructor Details

    • AIGenerationTask

      public AIGenerationTask()

  • Method Details

    • copy

      public AIGenerationTask copy()
      Creates a deep copy of the task.

    • clone

      protected Object clone() throws CloneNotSupportedException
      Deprecated.
      use copy()
      We override this to allow cloning for use with the copy() method, but deliberately leave it protected since the use of copy() is intended.
      Overrides:
      clone in class Object
      Throws:
      CloneNotSupportedException

    • maxTokens

      public AIGenerationTask maxTokens(Integer maxTokens)

    • addOptionalInputFile

      public AIGenerationTask addOptionalInputFile(@Nullable File file)

    • addInputFiles

      public AIGenerationTask addInputFiles(List<File> files)

    • addInputFile

      public AIGenerationTask addInputFile(File file)

    • setOutputFile

      public AIGenerationTask setOutputFile(@Nonnull File file)

    • embedComment

      protected String embedComment(String content, String comment)

    • hasToBeRun

      public boolean hasToBeRun()

    • calculateAllInputsMarkers

      @Nonnull protected List<String> calculateAllInputsMarkers()

    • determineFileVersionMarker

      protected String determineFileVersionMarker(@Nonnull File file)

    • shaHash

      protected String shaHash(String content)

    • getRecordedOutputVersionMarker

      protected AIVersionMarker getRecordedOutputVersionMarker()
      Version of current output file.

    • addPrompt

      public AIGenerationTask addPrompt(@Nonnull File promptFile, String... placeholdersAndValues)
      The actual prompt to be executed. The prompt file content can contain placeholders that are replaced by the values given: placeholdersAndValues contain alternatingly placeholder names and values for them.
      Returns:
      this

    • addPrompt

      public AIGenerationTask addPrompt(@Nonnull File promptFile, Map<String,String> placeholdersAndValues)
      The actual prompt to be executed. The prompt file content can contain placeholders that are replaced by the values given.
      Returns:
      this

    • getFileContent

      protected String getFileContent(@Nonnull File file)

    • unclutter

      protected static String unclutter(String content)

    • setSystemMessage

      public AIGenerationTask setSystemMessage(@Nonnull File systemMessageFile)

    • relativePath

      protected String relativePath(@Nullable File file, @Nonnull File rootDirectory)

    • execute

      public AIGenerationTask execute(@Nonnull Supplier<AIChatBuilder> chatBuilderFactory, @Nonnull File rootDirectory)
      Execute the task if necessary. If the output file is already there and up to date, nothing is done.

    • toJson

      public String toJson(@Nonnull Supplier<AIChatBuilder> chatBuilderFactory, @Nonnull File rootDirectory)
      For debugging purposes: returns the JSON that would be sent to the AI.

    • makeChatBuilder

      @Nonnull protected AIChatBuilder makeChatBuilder(@Nonnull Supplier<AIChatBuilder> chatBuilderFactory, @Nonnull File rootDirectory, String outputRelPath)

    • explain

      public String explain(@Nonnull Supplier<AIChatBuilder> chatBuilderFactory, @Nonnull File rootDirectory, @Nonnull String question)
      Ask a question about the previous task execution. We assume it was previously run (hasToBeRun() == false), add the result of the previous execution to the chat, and ask the AI the given question about it. This can be used e.g. to see why the AI did something, or in the process of improving the prompt, etc.
      Returns:
      the answer of the AI - not written to a file!

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • force

      public void force(boolean force)
      If true the generation will be run even if not hasToBeRun().