⚠️ Migration to the new events API #3262

clebs · 2025-07-22T20:32:25Z

This Pull Request migrates from the old events API (k8s.io/client-go/tools/record) to the new API (k8s.io/client-go/tools/events).

Closes #2141

k8s-ci-robot · 2025-07-22T20:32:35Z

Hi @clebs. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

clebs · 2025-07-22T20:36:47Z

Open Items:

StartRecordingToSink: manage the context/channel in the provider on ensureRecording and Stop.
StartWatcher: see if this is still needed and how to adapt it if it is.
Leader election: resourcelock.ResourceLockConfig is still using the old API.
This has now been solved by adding an adapter that turns old API calls into new API calls (albeit pretty simplistic). So that way on the backend we have only the new API and places that require the old can use the adapter.
Update unit tests.

alvaroaleman · 2025-07-22T20:50:59Z

/ok-to-test

alvaroaleman

two questions

pkg/recorder/recorder.go

alvaroaleman · 2025-07-23T20:40:09Z

pkg/recorder/recorder.go

+	GetEventRecorder(name string) events.EventRecorder
+	// GetOldEventRecorder returns an EventRecorder for the old events API.
+	// The old API is not 100% supported anymore, use the new one whenever possible.
+	GetOldEventRecorder(name string) record.EventRecorder


Rather than extending the interface here, would it be possible to build a wrapper that makes a record.EventRecorder out of a events.EventRecorder?

Yes, it is already built as a wrapper under the hood in intrec. I added it here with the idea to support both APIs for some time. If it is not what we are going for, having the wrapper only where it is needed is better.

I think there are two main ways we could go about this:

We make this a non-breaking change by keeping the existing GetEventRecorderFor with the existing signature and add a new GetEventRecorder with the new events.EventRecorder as return but mark the GetEventRecorderFor as // deprecated. At some point in a future release we will then remove GetEventRecorderFor

We make this a breaking change by changing the signature of the existing GetEventRecorderFor to return an events.EventRecorder and we provide an adapter somewhere to go from events.EventRecorder to a record.EventRecorder for places like the leader election

My vote if this is possible would be to do 1) as this is the least disruptive for c-r users but its possible I've missed something and it isn't possible

Option 1 is doable, but I have to see if the wrapper approach is good enough. Reason being, that the wrapper calls the new API under the hood and both APIs are quite different. The old API has AnnotatedEventf while the new one does not and the new one has things like relatedObject and action.

Otherwise, the other way to support both is having 2 distinct implementations in separate packages and duplicate the broadcaster, provider, interfaces and injection at the manager and cluster levels.
Finally also duplicate the tests so both versions are covered.

I will test if the wrapper is viable, present the results and then we can make an informed decision.

I have tested out the wrapper and it has the one downside: we loose the ability to set annotations on events since there is no equivalent AnnotatedEventf. I also have tried to get some info on why that is the case.

Seeing that, the wrapper can not provide the same features as the old API has, so the only option to support both is probably having both implementations at the same time.

@alvaroaleman my proposal is to duplicate the internal events package and have both implementations separate from each other even if that means lots of duplication for some time. It will make it easier later on to cleanup the old API when removed. What do you think?

@alvaroaleman my proposal is to duplicate the internal events package and have both implementations separate from each other even if that means lots of duplication for some time. It will make it easier later on to cleanup the old API when removed. What do you think?

Not great, but given its for a limited time only that seems okay to me

I am also not a great fan.

I will do the exercise and see how much I can reuse without having a tangled nightmare.

Worst case it is all duplicate.

@alvaroaleman I have gone through this and now this PR supports both the old and new events APIs with as little duplication as possible.

There is still a fair amount of duplication but it is confined mostly to internal code and should still be relatively easy to cleanup later on.

If you agree with this approach I will move to adding tests for the new API and move this out of draft.

If you agree with this approach I will move to adding tests for the new API and move this out of draft.

Sounds good to me

Tests added and moved out of draft. PTAL!
Still finishing up the recorder integration test for the new API, but the code should be ready for review.

pkg/internal/recorder/recorder.go

sbueringer · 2025-08-07T16:39:10Z

/assign

k8s-ci-robot · 2025-08-11T23:02:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: clebs
Once this PR has been reviewed and has the lgtm label, please ask for approval from sbueringer. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/internal/recorder/recorder.go

sbueringer · 2025-08-12T12:10:26Z

pkg/leaderelection/leader_election.go

@@ -128,7 +128,7 @@ func NewResourceLock(config *rest.Config, recorderProvider recorder.Provider, op
 		coordinationClient,
 		resourcelock.ResourceLockConfig{
 			Identity:      id,
-			EventRecorder: recorderProvider.GetEventRecorderFor(id),
+			EventRecorder: recorderProvider.GetEventRecorderFor(id), //nolint:staticcheck


What would be the plan to resolve this one?
Improve the upstream package so it can handle the new recorder, then use it here?

(if yes, let's make sure we have corresponding follow-up issues in CR & k/k)

Correct, this would require client-go to migrate to the new API too. I will make sure to create followups once this is in.

Perfect, thx!

pkg/manager/manager.go

pkg/recorder/recorder.go

sbueringer · 2025-08-20T14:00:37Z

Looks like it's still occuring (at least the linter caching issue is "fixed" now)

clebs · 2025-08-23T13:20:03Z

@sbueringer I indeed have to correct my previous statement saying it consistently succeeds. It does succeed when running make clean && make test, but when running hack/ci-check-everything.sh, as you suggested, it does fail.

I will look into this and try to get it fixed on CI. It seems the current state makes the tests timeout in different places.

sbueringer · 2025-08-25T09:03:44Z

/retest

(last one was an unrelated flake)

sbueringer · 2025-08-25T15:41:35Z

/test pull-controller-runtime-test

(Not sure if it should be already fixed or not :))

clebs · 2025-08-25T19:47:10Z

I have not made any changes since my last comment where I just rebased.

I ran ci-check-everything.sh for a while and it is flaky; Sometimes it succeeds and other times it fails with a variety of unrelated tests failing (e.g. manager, controller, cache, certwatcher...). But I observe the same flakiness on main, at least locally.

So I have been unable to pin this down and am not sure anymore whether it is still related to this PR.

sbueringer · 2025-08-26T06:03:06Z

Last time I checked I didn't see flakes like this on main in testgrid. Also didn't notice on other PRs. Maybe the changes here make the flake more likely

Edit: https://storage.googleapis.com/k8s-triage/index.html?pr=1&text=found%20unexpected%20goroutines&job=.*controller-runtime.* (4 occurences, all on this PR) Not sure if I'm searching for the right failure message

clebs · 2025-08-26T16:30:33Z

Last time I checked I didn't see flakes like this on main in testgrid. Also didn't notice on other PRs. Maybe the changes here make the flake more likely

Edit: https://storage.googleapis.com/k8s-triage/index.html?pr=1&text=found%20unexpected%20goroutines&job=.*controller-runtime.* (4 occurences, all on this PR) Not sure if I'm searching for the right failure message

Thanks for the infos @sbueringer ! It has helped me focus on this one issue and I found the leaking goroutine I think (on my potato laptop I get lots of noise from other failures when testing thus hard to confirm 100%).

TLDR; I was ignoring the stopFunc returned by StartEventWatcher(), now this is called when the broadcaster is shutdown.

sbueringer · 2025-08-26T17:35:27Z

Very nice that the test found an actual issue :)

/test pull-controller-runtime-test

(I'm going to run the test a few more times just for more data)

sbueringer · 2025-08-26T18:17:01Z

Damn :/

I'm starting to lean towards intentionally not merging this into the current minor release, even if the tests are fine a few times in a row eventually, because I would prefer data from testgrid over a longer period to ensure we're not introducing an issue

clebs · 2025-08-26T18:43:15Z

Damn :/

I'm starting to lean towards intentionally not merging this into the current minor release, even if the tests are fine a few times in a row eventually, because I would prefer data from testgrid over a longer period to ensure we're not introducing an issue

Agreed, this needs more effort.
The goroutine from the watcher was definitely leaking since we were never stopping it but there seems to be more to it.

I am not able to properly work on this with my current laptop. I will resume testing when I have access to modern hardware in a few days.

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

The StartEventWatcher call is no longer needed since calling StartRecordingToSink already sets up a watcher and manages it via its context. Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

- Remove outdated comment - Improve error handling when setting default Options - Replace deprecated sink recording function - log errors from StartEventWatcher Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2025

k8s-ci-robot requested review from JoelSpeed and sbueringer July 22, 2025 20:32

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 22, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 22, 2025

clebs changed the title ~~⚠ Migration to the new events API~~ ⚠️ Migration to the new events API Jul 22, 2025

clebs mentioned this pull request Jul 22, 2025

Switch to use events.EventRecorder instead of record.EventRecorder #2141

Open

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 22, 2025

clebs force-pushed the new-events-api branch from 732ed05 to c8698d8 Compare July 23, 2025 15:49

alvaroaleman reviewed Jul 23, 2025

View reviewed changes

clebs force-pushed the new-events-api branch 2 times, most recently from 6bfbd83 to 0528c33 Compare July 24, 2025 12:54

valdar reviewed Jul 25, 2025

View reviewed changes

pkg/internal/recorder/recorder.go Outdated Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 29, 2025

clebs force-pushed the new-events-api branch from 592ade2 to 4541c71 Compare July 30, 2025 14:02

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 30, 2025

clebs marked this pull request as ready for review August 7, 2025 09:26

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 7, 2025

k8s-ci-robot requested review from alvaroaleman and troy0820 August 7, 2025 09:26

clebs force-pushed the new-events-api branch from b1e2d6c to bc36d51 Compare August 7, 2025 09:31

k8s-ci-robot assigned sbueringer Aug 7, 2025

sbueringer reviewed Aug 12, 2025

View reviewed changes

clebs force-pushed the new-events-api branch from 48a805d to a1ca95e Compare August 23, 2025 13:38

clebs force-pushed the new-events-api branch from fa81bdd to 4f82545 Compare August 28, 2025 07:53

clebs added 17 commits September 10, 2025 17:41

Base migration to the new events API

4598923

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Add support for both the new and old events APIs simultaneously.

b8dfa5b

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Adapt StartEventWatcher to the new API

59f7a1a

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Revert renaming of GetEventRecorderFor

145585f

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Remove StartEventWatcher call

18fb57c

The StartEventWatcher call is no longer needed since calling StartRecordingToSink already sets up a watcher and manages it via its context. Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Fix linting issues

322b8ff

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Improve message when no action is provided for an event.

5cafd3d

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Support both events APIs

b2027c4

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Add unit tests for the new events API

19eb81d

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Fix new API not broadcasting events and add integration test

86a2a84

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Apply review feedback

9e1910d

- Remove outdated comment - Improve error handling when setting default Options - Replace deprecated sink recording function - log errors from StartEventWatcher Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Log errors from StartRecordingToSinkWithContext

6563300

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Return early if starting to record fails

78dcc4b

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Fix new events API manager leak detection test

8e73a31

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Fix recorder integration test

68da697

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Fix duplicate recorder name

d97bf91

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

Stop event watcher when broadcaster is shutdown

9a5f2a6

Signed-off-by: Borja Clemente <borja.clemente@gmail.com>

clebs force-pushed the new-events-api branch from 4f82545 to 9a5f2a6 Compare September 10, 2025 15:41

⚠️ Migration to the new events API #3262

Are you sure you want to change the base?

⚠️ Migration to the new events API #3262

Conversation

clebs commented Jul 22, 2025

Uh oh!

k8s-ci-robot commented Jul 22, 2025

Uh oh!

clebs commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alvaroaleman commented Jul 22, 2025

Uh oh!

alvaroaleman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alvaroaleman Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebs Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebs Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbueringer commented Aug 7, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Aug 20, 2025

Uh oh!

clebs commented Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbueringer commented Aug 25, 2025

Uh oh!

sbueringer commented Aug 25, 2025

Uh oh!

clebs commented Aug 25, 2025

Uh oh!

sbueringer commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clebs commented Aug 26, 2025

Uh oh!

sbueringer commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbueringer commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clebs commented Jul 22, 2025 •

edited

Loading

alvaroaleman Jul 23, 2025 •

edited

Loading

clebs Jul 24, 2025 •

edited

Loading

clebs Aug 7, 2025 •

edited

Loading

clebs commented Aug 23, 2025 •

edited

Loading

sbueringer commented Aug 26, 2025 •

edited

Loading

sbueringer commented Aug 26, 2025 •

edited

Loading

sbueringer commented Aug 26, 2025 •

edited

Loading