Aki Hamano

How to Collaborate with AI When Contributing Code to WordPress

,

This is not limited to WordPress, but as AI advances, I feel that development is getting faster and, at the same time, new contributors can get involved in code contributions more easily than before.

On the other hand, I would like to list some things I personally want to be careful about and uphold when using AI for contributing.

Writing code is a means, not an end

You submit a pull request, it gets reviewed, and it gets merged. Then it is treated as measurable props, and your name appears in the credits for the next WordPress release. And if it is your first contribution, a “Core Contributor” badge will also be added to your wordpress.org profile page. Having your contributions remain in a visible form is, I think, one of the biggest motivations for continuing to contribute.

But hasn’t obtaining that kind of “proof” — that is, writing code, submitting a PR, and having it merged — become the goal in itself?

If you just tell an AI “submit a PR that fixes this issue” or “implement the feature proposed in this issue,” the AI will immediately implement it in code and auto-generate the PR description. But before that, I think you should consider the following.

  • If the issue is a bug report:
    • Can that bug really occur in the real world or through human operation? If it cannot realistically occur, or cannot be reproduced manually, it might be closed as “won’t fix.”
    • Is that bug really a problem to be solved on the WordPress side? Isn’t it something the plugin should handle?
    • Do you really want to fix that error? Won’t you end up silently swallowing an error that should be raised?
    • Is fixing that error enough? Isn’t there a larger, more fundamental bug lurking upstream?
  • If the issue is about a feature request:
    • Can’t that feature be achieved by combining existing features?
    • Is that feature worth maintaining forever? WordPress is an OSS project that greatly values backward compatibility, so removing an API once it has been added is difficult.
    • Would more than 80% of users want to use that feature? Should that feature really be added to core?
    • Is it really ready to be added? Has there been enough discussion among contributors about adding that feature?
    • And above all, do you yourself feel that the feature has value? Can you explain that value in your own words?

I am not saying you must not submit PRs. Precisely because AI has made writing code easy, I think questioning things and forming hypotheses before submitting a PR has become more important.

As a result, the need to submit a PR at all may disappear, and the issue may be closed. Unfortunately, that may not be counted as props as data, but I personally think it is a more valuable contribution than a PR submitted without any thought.

The goal is to think about what should be done to make WordPress better, and to discuss it with other contributors. Writing code and submitting PRs are merely means to achieve that; they are not the goal.

Aim for human-readable comments

When you ask an AI to generate content in order to report some bug, it may generate extremely detailed content like the following.

Symptoms

When XXX is executed in the state of XXX, XXX occurs. This phenomenon reproduces only when XXX, and does not reproduce when XXX. Furthermore, because it occurs without any XXX or XXX whatsoever, XXX cannot notice that this defect is occurring at all, and XXX likewise cannot notice that it is being shown a XXX that differs from XXX. That is, the symptom is that XXX occurs in a completely silent manner for both the XXX side and the XXX side.

Expected result

Regardless of what value XXX is set to, XXX is expected to always be XXX at least in XXX, and XXX is expected to be XXX as intended. Alternatively, if that is technically difficult, then at the point where a XXX is passed that would be judged valid only by a XXX depending on XXX, a warning is expected to be reported to XXX, as consistent behavior in line with how XXX already fires XXX for XXX. In any case, the current outcome of breaking silently with no notification at all is not what is expected.

Cause

The cause lies in a mismatch of assumptions between three parties: XXX, defined on lines XX through XX of the XXX.js file; XXX, which calls it within the same file; and XXX, which assembles XXX on lines XX through XX of the XXX.js file.

Specifically, XXX converts XXX into XXX using the fixed value XXX as the reference, via the single line XXX on line XX. And this conversion result is used solely for internal XXX. However, the XXX that is actually output is generated without going through this conversion, retaining the original XXX written by XXX as-is. This point is explicitly described as the implementation intent in the comment spanning lines XX through XX.

The problem here is the behavior in the XXX specification whereby XXX written inside XXX is resolved against XXX rather than XXX. In other words, even if the XXX side, on the premise of the fixed value XXX, determines that “XXX is smaller than XXX,” that determination is not reproduced on the XXX side. When XXX is set larger than XXX, only the effective value on the XXX side increases while the XXX side does not change, so the magnitude relationship between the two is inverted. Once the magnitude relationship is inverted, the lower bound of the XXX assembled in the form of XXX exceeds its upper bound, and as a result XXX becomes logically XXX and matches no XXX at all. Once it no longer matches, the XXX and XXX output inside it are not applied at all, and XXX ends up XXX in situations where it should not be, or the XXX that should be applied remains unapplied.

Note that when XXX is written using the same XXX, both change at the same ratio even if XXX changes, so the magnitude relationship is preserved and this problem does not occur. Also, specifying XXX cannot work around this problem, because XXX inside XXX is resolved against XXX rather than XXX.

Proposed fix

By fixing XXX on line XX of the XXX.js file so that it is consistent with the processing on the XXX side, the value used for XXX and the value actually output will match, and any combination that passes XXX will necessarily be a valid XXX on XXX as well. That is, we adopt the consistent policy of also using the result of converting XXX into XXX for the XXX output. Since the criterion XXX is already adopted on the XXX side, this policy does not introduce a new premise into the system as a whole; it merely applies an existing premise to the output side as well, so there is no concern that XXX would be impaired.

However, since this policy effectively reduces XXX to XXX, it may change the intent of XXX who deliberately chooses XXX. As an alternative, therefore, one could also consider a policy of requiring XXX to be identical within XXX, and treating a mixture of XXX as an invalid value. As a third alternative, one could consider continuing to accept XXX as before, while firing XXX only in cases where that XXX depends on the premise of XXX, and additionally documenting the constraint regarding XXX in the XXX documentation. The third policy does not change the existing XXX at all, so it is the safest from a backward-compatibility standpoint.

Test steps

  1. Prepare a verification environment with XXX or later installed, and activate XXX.
  2. Edit XXX, add a setting with mixed XXX directly under XXX as shown below, and save. XXX
  3. In an environment where XXX is available, run the following command and check the contents of the generated XXX. XXX
  4. Confirm that the value corresponding to XXX in the output is XXX — that is, that the XXX on the XXX side is retained as-is without being converted.
  5. Create XXX, place a single XXX, then set XXX on it and publish.
  6. Open XXX from the XXX settings screen, and change XXX to XXX.
  7. Adjust XXX to any value between XXX and XXX, then display the XXX published in step 5.
  8. Confirm that, because XXX is XXX, XXX is resolved as XXX, the generated XXX becomes a XXX whose lower bound exceeds its upper bound, and the XXX set in step 5 is not applied — that is, that XXX which should not be XXX remains XXX.
  9. As a control experiment, change the value in step 2 from XXX to XXX so that XXX is identical, then run steps 6 through 8 again, and confirm that in this case XXX is applied correctly, and that the behavior does not change even when switching XXX between XXX, XXX, and XXX.
  10. After applying the fix, restore the setting from step 2, run steps 6 through 8 again, and confirm — depending on the fix policy adopted — that XXX no longer becomes XXX and XXX is applied, or that the corresponding warning is output to XXX when XXX is enabled.

The first thing I think when I see this is: “So what?“

The content may contain not a single contradiction and may be entirely correct technically, but the ones who work on this issue and discuss it here are humans. In the future AI may handle all of it, but at least in WordPress development and contribution today, that is not the case.

Some contributors may be pasting AI research results directly into their content with the best of intentions, but the problem is not whether AI was used to generate the content. I think the following should be observed:

  • Did you review and proofread all of the generated content yourself?
  • Did you make an effort to summarize it and remove redundant parts, to reduce the burden on the humans who will read it?
  • Did you really carry out those test steps yourself?
  • Is the fix the AI proposes really valid? And shouldn’t the fix be decided through discussion in the first place?
  • Above all, do you yourself find that content easy to read?

Code review

Recently I have seen several submitted PRs with comments like the following added to them.

This review is not based on running XXX, but on static analysis after reading XXX.

As my understanding of the changes, I understand that this PR makes the following X changes:

  • Around line XX of XXX.js, changing processing that previously targeted only XXX so that it also targets XXX
  • Adding a new XXX on lines XX through XX of the same file
  • Adding X tests to XXX.spec.js that verify XXX

Good points include the following:

  • The implementation approach for XXX is appropriate. A naive implementation would tend to use XXX, but that can only obtain XXX. This PR goes through XXX, which reliably makes the processing proceed in the direction of XXX.
  • The match of XXX is guaranteed. Line XX of XXX.php and line XX of XXX.php both derive XXX from the same value, so I confirmed they are correct as comparison targets.
  • Because XXX is called, XXX does not occur.

Issue (Blocking). Line XX of XXX.js throws XXX when XXX contains XXX. Because this PR makes this processing callable from XXX as well, an exception may be thrown during XXX, potentially making the entire XXX XXX. I propose wrapping it in try / catch and falling back to XXX on failure.

Note that XXX contains the same code, so this is not a problem introduced by this PR; however, since it becomes newly reachable, I think it is desirable to address it within the scope of this PR.

Issue (Non-blocking). The X added tests only verify the new path. Meanwhile, the XXX >= 0 branch is existing behavior, yet it is modified by this PR and is not verified. The original issue also mentions the XXX case, so I think it would be desirable to assert that as well.

Issues (Nit / optional), the following 3 points:

  • XXX on line XX of XXX.js duplicates line XX. There is room to consolidate them in one place, though there is no problem with leaving it as-is.
  • The optional chaining XXX?.XXX?.XXX does not function in this context.
  • XXX does not move after XXX. The same applies to XXX, so a follow-up may be appropriate, but I think it is worth clarifying whether this is an intentional decision.

In summary, the direction is sound, and the implementation of XXX in particular is commendable. I think the one Blocking point needs to be addressed before merging. Please also consider the one Non-blocking point.

These are very accurate observations, and if this PR is merged, it may be counted as a contribution by the person who posted the comment — that is, as props. But if this comment is simply the result of asking an AI to “review this PR,” and the poster has not scrutinized the comment at all themselves, can it really be called a contribution by that poster? Personally, I do not think so.

Reviews by AI tools are extremely powerful and convenient, but in WordPress at present, AI must not be the sole reviewer.

  • Do you understand the review the AI generated, and can you explain why that change is necessary?
  • If the PR is updated based on that review, can you ultimately take responsibility for approving that PR with confidence?
  • Is that review easy for humans to read? Rather than putting everything into one comment, wouldn’t it be clearer to split the review into inline comments on the code, make use of suggestions, or attach videos and images?

Summary

Now that the advance of AI has made it easier to contribute code to WordPress, I think the important things come down to these two:

  • Always question things and form hypotheses
  • Be conscious that the other party is a human

Leave a Reply

Your email address will not be published. Required fields are marked *