Build Speed Improvement fast follow fixes

[williscool]

Needed Now

• fix the coverage reports (we need to bring back the classes file stuff as artififcats for that to work
• use the dorny reporter conclusion thing to fail (our running script only runs we don't want it to fail the build because then we might not get coverage. which we always want)
• there should be 3 caches. the instrument tests should have a different cache from the unit tests and from build. they all do different things now and there should be less overlap. that should cut down load times on unit and instrument tests also
• delete the old bad caches for feat!: dismiss from reschedule confirmations #73
• after breaking and then fixed connectedAndroidTest we should have a seperate job like the test sumarray one that kicks off at the same time as the new faster buid that runs wait for emulator and just ./gradlew cAT -Pandroid.testInstrumentationRunnerArguments.class=com.github.quarck.calnotify.calendarmonitor.ComponentIsolationTest --parallel --max-workers=$(nproc) --build-cache --configure-on-demand which is enough to make sure its not broken so we can go back to it when we need (i.e is what android studio uses and its great in dev because you don't have to remember to build the app again after you make a change it just does it for you
• we should also evaluate just doing cat in the first place and signing after that. Part of me wonders if discounted that too much and underestimated replacing it let's just do them in parallel for a bit and see
• dorny reporter summary should end up on the summary output but its not for some reason (its getting made as you can see in the log output but not put on the output page in actions)
• the seperate test summary job is borked because I changed the reports to have the build time in them

---

future

• I think there is a bug in the test listener where if the tests errror (on like an intermittent failure with the mocks) it fails to report. just make sure that isn't happening (verified the bug 4-22-2025) see Build Speed Improvement fast follow fixes #77 (comment) below
• fix the android cache. and other caches so we dont have misses anymore (i.e. i'm seeing cache misses on ccache on the first run of new prs sometimes when it would be most beneficial)
• why are we redownloading the android sdk everytime for the emulator?

Originally posted by @williscool in #76 (comment)

[williscool]

dorny report already works with failures!

[williscool]

Verified issue with test listener. sumarries from my (and GPT 4.1s) investigation

---

Context from Initial Investigation

• The issue was observed when running instrumentation tests, where if a test errors (such as from an intermittent failure or a mock not working correctly due to test order), the results file is empty.
• Example logs showed a fatal exception (NullPointerException) during test execution, followed by normal test output, but ultimately no test results XML was found or the file was empty.
• Coverage data was still generated correctly, but the absence of the XML results file led to warnings and CI failures.
• This behavior was confirmed to be due to the XML report only being written if the test run completes normally, and not in the event of a process crash or fatal error.

---

Findings: Instrumentation Test XML Results File Empty on Crash

Problem

• When an instrumentation test run crashes (e.g., due to a NullPointerException or other fatal error), the XML results file is not written or is left empty.
• This is because the XmlRunListener only writes the XML report in the testRunFinished callback, which is not called if the process crashes or is killed.
• As a result, CI systems (like GitHub Actions) see an empty results file and treat the run as a failure, which is the correct behavior for now.

Cause

• The Android instrumentation test process does not guarantee that listeners (like XmlRunListener) will be notified if the process is killed or crashes.
• The code that writes the XML report (printTestResults()) is only called in testRunFinished, so if the process dies unexpectedly, no report is written.

Workarounds & Next Steps

• For now, the CI system correctly treats empty results as a failure.
• A possible future improvement is to create a fallback reporter that parses stdout to reconstruct results if coverage succeeds and all tests pass but the XML report is missing.
• No changes to production code are needed at this time.
• This limitation is common to most JUnit-based Android test runners, not just this project.
• If you see an empty or missing XML report, always check the logs for fatal exceptions or process crashes.
• If you implement a fallback reporter in the future, document its limitations (e.g., it may not capture all test metadata).
• For debugging, saving the raw instrumentation output (stdout) can help reconstruct what happened during a failed/crashed run.
• If you need more reliable reporting, consider running tests in smaller batches or using the Android Test Orchestrator, which can help isolate failures.

Inital context from investigation

this is a bug in the running of the instrument tests where if a test errors (say from like an intermittent failure from a mock not working correctly because of a test order thing)

the results file is empty can you find it?

example logs

Running instrumentation tests with 30m timeout...
--------- beginning of main
04-22 21:34:18.003  2886  2886 D AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 2000 <<<<<<
04-22 21:34:18.023  2886  2886 I AndroidRuntime: Using default boot image
04-22 21:34:18.023  2886  2886 I AndroidRuntime: Leaving lock profiling enabled
04-22 21:34:18.276  2886  2886 D AndroidRuntime: Calling main entry com.android.commands.am.Am
04-22 21:34:20.662  2977  2977 D AndroidRuntime: Shutting down VM
--------- beginning of crash
04-22 21:34:20.664  2977  2977 E AndroidRuntime: FATAL EXCEPTION: main
04-22 21:34:20.664  2977  2977 E AndroidRuntime: Process: com.github.quarck.calnotify, PID: 2977
04-22 21:34:20.664  2977  2977 E AndroidRuntime: java.lang.NullPointerException: Attempt to invoke virtual method 'android.content.res.Configuration android.content.res.Resources.getConfiguration()' on a null object reference
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.app.ConfigurationController.updateLocaleListFromAppContext(ConfigurationController.java:266)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6880)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.app.ActivityThread.-$$Nest$mhandleBindApplication(Unknown Source:0)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2236)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:106)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.os.Looper.loopOnce(Looper.java:205)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:294)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:8177)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
04-22 21:34:20.664  2977  2977 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
INSTRUMENTATION_STATUS: class=com.github.quarck.calnotify.calendar.CalendarBackupRestoreTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=49
INSTRUMENTATION_STATUS: stream=
com.github.quarck.calnotify.calendar.CalendarBackupRestoreTest:
INSTRUMENTATION_STATUS: test=testFindMatchingCalendarId_ExactMatch
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.github.quarck.calnotify.calendar.CalendarBackupRestoreTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=49
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=testFindMatchingCalendarId_ExactMatch

... rest of the tests


INSTRUMENTATION_STATUS: class=com.github.quarck.calnotify.calendarmonitor.SimpleCalendarMonitoringTest
INSTRUMENTATION_STATUS: current=49
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=49
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=testSimpleCalendarMonitoring
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: coverageFilePath=/data/data/com.github.quarck.calnotify/files/coverage.ec
INSTRUMENTATION_RESULT: stream=

Time: 157.48

OK (49 tests)

... coverage generated correctly

Pulling chunk 140/140 (offset: 142336)...
Method 3 result: Local file size = 143192 bytes
Setting up coverage data for JaCoCo...
File exported to: ./app/build/outputs/code_coverage/X8664DebugAndroidTest.ec
JaCoCo coverage file size: 143192 bytes
✅ Coverage data prepared successfully! (140K bytes)
You can now run JaCoCo report generation with standard Gradle commands:
  cd android && ./gradlew jacocoAndroidTestReport
Coverage data preparation completed
Processing XML test results for test reporting...
Attempting to pull XML test results directly...
✅ Successfully pulled test results directly!
⚠️ No test results XML found or file is empty.

[williscool]

I think we should move back to cAT

But use robo electric also

CAT will run on all prs but only block releases if they fail

For prs will allow merges to master with only robo electric passing

[williscool]

Resolved via

#85

#94

And

#97