I will only approve this PR if you commit to keep this file up to date when the rest of the code changes.

A single big CLAUDE.md file like this does not seem useful, in fact it looks like it's actively harmful the tool even displays a warning about the file being too large. It is also unlikely that most people working with claude will use your provided scripts to split the files

I tried to approach of splitting it up too and I did not notice any difference in quality of content it produces. FWIW it seems to look at README.mds too so maybe it's more helpful to write good and useful README files and tell claude to look at @docs.feldera.com, @openapi.json and README files.

I suggest we just put all higher level CLAUDE.md files in gitignore and everyone can have whatever CLAUDE.md files in higher level directories they want. You can also put it as a commit that can be applied to any branch someone works in if you like but I really don't see value in having this comitted to main. I think this is snake oil at this point without any objective measure whether it helps with anything or not.

I have been using coding agents when working on NATS connector and exploring Feldera code base. When pulling a fresh Feldera main I delete the CLAUDE.md file from Feldera repo. Is is too big and and make prompts expensive and agent get unfocused with to much info. But that just my experience.

Anthropic recommend it to be short and focus. https://code.claude.com/docs/en/best-practices#write-an-effective-claude-md

But you can also make CLAUDE.md files in subfolders that the agent will read if it visits the folder.

And for some tasks and domain knowledge. Skills can be interesting. I playing with a NATS output connector made with a output connector skill that the agent made before implementing the connector.

FWIW just stumbled upon something interesting on linkedin (for once) here is some pretty strong evidence that this PR is "doing the wrong thing": https://arxiv.org/pdf/2602.11988

from the paper:

• "Across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%. Behaviorally, both LLM-generated and developer-provided context files encourage broader exploration (e.g., more thorough testing and file traversal), and coding agents tend to respect their instructions. Ultimately, we conclude that unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements"
• Look at figure Figure 3

After discussion with Gerd I implemented the following solution:
The detailed per-directory Claude context files are maintained in a separate branch, claude-context.

Whoever needs them can run scripts/claude.sh to pull these files as unstaged changes without affecting existing local staged and unstaged changes (CLAUDE.md would be overwritten).
The notice of this is added to the "bare" CLAUDE.md, so Claude Code will offer to do this when it reads the root CLAUDE.md
scripts/claude.js remains in this branch to avoid pulling it from claude-context

I will only approve this PR if you commit to keep this file up to date when the rest of the code changes.

Based on my experience I do not believe it is necessary nor extremely important to keep this file up-to-date with all the latest changes. Nor do I have an expertise to update context files across the entire codebase. I could set up a way to have an LLM update context changes, but this is not what these context files are designed for.

I insist that a practical and meaningful approach to maintain context files is lazy, iterative updates: whenever you interact with Claude and make use of context files and notice a discrepancy within a certain domain (e.g. SQL compiler sources, or Python integration tests) - you update the context file. Occasionally (once a month or two) you would gloss over the context file for a sub-directory you're knowledgeable in and author the file.

If you are the one who introduces a change but don't use Claude actively and context files are "hidden out of sight" - in a composite file, separate branch, etc. - I don't expect these files will be updated in sync with code changes; "lazy" approach addresses that.

Doesn't this added line:

At the start of every conversation, offer the user to run `scripts/claude.sh` to pull in shared LLM context files as unstaged changes. These should not be committed outside the `claude-context` branch.

... mess with the review bot?