RFC notifications bot (Dec 10 2020 at 23:52): RFC notifications bot (Dec 10 2020 at 23:52):fitzgen updated PR #4 from benchmark-suite to main:
Rendered RFC
Note: this is complimentary to RFC #3. Where that RFC is seeking consensus on whether we should establish benchmarking infrastructure at all, this RFC focuses on defining where we want our benchmarking story to end up at the end of the day: how we select benchmark programs, how we avoid measurement bias, how we perform sound and rigorous statistical analyses of the results, etc.
RFC notifications bot (Dec 11 2020 at 00:02):fitzgen updated PR #4 from benchmark-suite to main:
Rendered RFC
Note: this is complimentary to RFC #3. Where that RFC is seeking consensus on whether we should establish benchmarking infrastructure at all, this RFC focuses on defining where we want our benchmarking story to end up at the end of the day: how we select benchmark programs, how we avoid measurement bias, how we perform sound and rigorous statistical analyses of the results, etc.
RFC notifications bot (Jan 11 2021 at 19:42):fitzgen updated PR #4 from benchmark-suite to main:
Rendered RFC
Note: this is complimentary to RFC #3. Where that RFC is seeking consensus on whether we should establish benchmarking infrastructure at all, this RFC focuses on defining where we want our benchmarking story to end up at the end of the day: how we select benchmark programs, how we avoid measurement bias, how we perform sound and rigorous statistical analyses of the results, etc.
RFC notifications bot (Jan 11 2021 at 19:42):fitzgen edited PR #4 from benchmark-suite to main:
Rendered RFC
Note: this is complimentary to RFC #3. Where that RFC is seeking consensus on whether we should establish benchmarking infrastructure at all, this RFC focuses on defining where we want our benchmarking story to end up at the end of the day: how we select benchmark programs, how we avoid measurement bias, how we perform sound and rigorous statistical analyses of the results, etc.
RFC notifications bot (Jan 11 2021 at 20:07):cfallin submitted PR Review.
RFC notifications bot (Jan 11 2021 at 20:11):alexcrichton submitted PR Review.
RFC notifications bot (Jan 11 2021 at 20:33):abrown submitted PR Review.
RFC notifications bot (Jan 11 2021 at 20:42):sunfishcode submitted PR Review.
RFC notifications bot (Jan 11 2021 at 20:42):sunfishcode submitted PR Review.
RFC notifications bot (Jan 11 2021 at 20:42):sunfishcode created PR Review Comment:
Reading through this RFC again, one thing that occurs to me is that it doesn't clarify whether "microbenchmarks" are in scope or not. On one hand, we might say that micro-benchmarks are in indeed in scope, specifically because we're not trying to build a General-Purpose benchmark suite, but instead just something to collect data to help identify performance changes over time. On the other hand, some parts of this proposal talk about a desire for a representative corpus. Could you clarify the intended stance on microbenchmarks?
RFC notifications bot (Jan 11 2021 at 20:42):sunfishcode created PR Review Comment:
Should this be enclosed in
[and]to linkify it?
RFC notifications bot (Jan 11 2021 at 21:19):fitzgen updated PR #4 from benchmark-suite to main:
Rendered RFC
Note: this is complimentary to RFC #3. Where that RFC is seeking consensus on whether we should establish benchmarking infrastructure at all, this RFC focuses on defining where we want our benchmarking story to end up at the end of the day: how we select benchmark programs, how we avoid measurement bias, how we perform sound and rigorous statistical analyses of the results, etc.
RFC notifications bot (Jan 11 2021 at 21:31):fitzgen submitted PR Review.
RFC notifications bot (Jan 11 2021 at 21:31):fitzgen created PR Review Comment:
I think my position is summarized in my response to Johnnie up thread:
Micro-benchmarks have their place, and we can include some as we find blind spots and want to target specific things (like Lars also mentioned) but I think it makes sense, especially as we create the initial set of candidates, to focus on real programs.
Multiple people have mentioned the desire to run microbenchmarks, and if we get value out of having them, I don't want to stand in the way for ideological reasons. As long as the microbenchmarks use the same benchmarking API/protocol that the rest of the corpus uses, supporting them should be straightforward and shouldn't require any additional engineering effort.
RFC notifications bot (Jan 22 2021 at 23:48):fitzgen merged PR #4.
Last updated: Nov 22 2024 at 16:03 UTC