Quick improvement suggestion: Reduce diff size with --ignore-all-space about aicommits HOT 6 CLOSED

smaller diff text:

const diffCached = ['diff', '--cached','--ignore-all-space','--diff-algorithm=minimal'];

from aicommits.

I thought about this, but some commits are just white-space changes (e.g. stylistic/linting).

If the diff doesn't show, GPT wouldn't be able to produce an accurate description of the change.

from aicommits.

Or actually even better, now that the new chat API is available and they take turns, the entire diff could be consumed file by file actually if detect it's bigger than some threshold and have a turn by turn chat, managing then to generate a comment about each and then asking to summarise each comment in the end. This would allow to work with quite large diffs as well.

https://github.com/openai/openai-python/blob/main/chatml.md
https://platform.openai.com/docs/guides/chat/chat-vs-completions

from aicommits.

I wonder if we can be smart about this and detect how much of the diff is white-space changes.

If the diff is mainly white-space change, we can pass in the diff with white-space.

If not (or if the diff is close to the OpenAI size limit), we can pass in the white-space ignored diff.

from aicommits.

It could be a conf option, but for many cases detecting these changes are important, particularly for languages where indentation matters and as above - ie formatting & linting changes. How about just counting the tokens with tokeniser and set it to the models max number instead of line count? and allow people to set a limit themselves.

from aicommits.

I don't think we can ignore white space for the reason provided above, and we've added --diff-algorithm=minimal so I think this is closable.

@salomartin
That idea sounds interesting but I worry that it may (unexpectedly) incur large expenses for generating a commit.

from aicommits.