What makes a bug discoverable?

Because agents receive no issue description, the expected behavior for each bug must be recoverable from the base repository or from a failure that is unambiguously wrong. These examples are drawn from the benchmark construction appendix.

Repository-defined contracts

A repository's documentation, types, tests, callers, identifiers, invariants, referenced standards, and mathematical definitions can establish expected behavior.

Accepted Python

Documentation · aiohttp

The API docs define the inherited status attribute as the HTTP response status.

  raise ContentTypeError(
      self.request_info,
      self.history,
+     status=self.status,
aiohttp PR #8745 ↗
Accepted Python

Docstring · pandas

The period_range docstring says freq is taken from a Period start or end; the code dropped its multiplier.

+     mult = freq.n

  if periods is not None:
      periods = periods * mult
pandas PR #53709 ↗
Accepted Python

Type annotation · sqlglot

Table.name is declared to return str, but an unset self.this made it raise AttributeError.

  @property
  def name(self) -> str:
-     if isinstance(self.this, Func):
+     if not self.this or isinstance(self.this, Func):
          return ""
      return self.this.name
sqlglot PR #4699 ↗
Accepted Python

Pre-existing test · SciPy

An existing test_weightedtau already asserts the NaN-in-y result that this copy-paste typo silently breaks.

  # If there are NaNs we apply _toint64()
  if np.isnan(np.sum(x)):
      x = _toint64(x)
- if np.isnan(np.sum(x)):
+ if np.isnan(np.sum(y)):
      y = _toint64(y)
SciPy PR #12955 ↗
Accepted Java

Caller or example · Netty

The caller adds the response to out to propagate it unreleased; this catch path wrongly released the still-forwarded response.

  } catch (Throwable t) {
-     release(response);
      ctx.fireExceptionCaught(t);
      removeThisHandler(ctx);
Netty PR #15602 ↗
Accepted Python

Identifier · Click

last_option, filled while scanning reversed(args), should retain the nearest option, but a missing break let an earlier one overwrite it.

      if _start_of_option(ctx, arg):
          last_option = arg
+         break

  return last_option is not None and last_option in param.opts
Click PR #2816 ↗
Accepted JavaScript

Internal invariant · Jest

fromJSON(toJSON(x)) must round-trip losslessly; a wrong array index corrupted the rebuilt duplicates map.

  private static mapFromArrayRecursive(
    arr: ReadonlyArray<[string, unknown]>,
  ): Map<string, unknown> {
-   if (arr[0] && Array.isArray(arr[1])) {
+   if (arr[0] && Array.isArray(arr[0][1])) {
      arr = arr.map(el => [
Jest PR #16353 ↗
Accepted Python

Parallel implementation · Dask

Sibling linspace and arange implementations coerce dtype=None to a default; eye omitted this and crashed.

  if M is None:
      M = N
+ if dtype is None:
+     dtype = float
Dask commit e25284d ↗
Accepted JavaScript

Referenced standard · Luxon

The RFC 2822/5322 grammar admits the UT zone token, but obsOffsets omitted it, so fromRFC2822 returned an invalid date.

  const obsOffsets = {
    GMT: 0,
+   UT: 0,
    EDT: -4 * 60,
    EST: -5 * 60,
Luxon PR #1782 ↗
Accepted Python

Mathematical definition · NumPy

The docstring defines var = mean(abs(x - x.mean())**2); squaring a complex array as x*x dropped the absolute value and produced a non-real variance.

  x = asanyarray(arr - arrmean)
- if issubclass(arr.dtype.type, nt.complexfloating):
-     x = um.multiply(x, um.conjugate(x), out=x).real
- else:
+ if issubclass(arr.dtype.type, (nt.floating, nt.integer)):
      x = um.multiply(x, x, out=x)
+ else:
+     x = um.multiply(x, um.conjugate(x), out=x).real
NumPy PR #13177 ↗

Repository-independent failure modes

A crash, hang, or data loss is unambiguously undesirable without a repository-specific contract. Such a failure is usually one of several sources of evidence; only the go-git race and OpenCV out-of-bounds read rely on this category alone.

Accepted Go

Crash · Go

A v holding a nil value fell through to reflectlite.TypeOf(nil).String(), causing a nil-pointer dereference and panic.

  func stringify(v any) string {
    switch s := v.(type) {
    ...
+   case nil:
+       return "<nil>"
    }
    return reflectlite.TypeOf(v).String()
Go commit 70e453b ↗
Accepted Java

Hang · OpenSearch

A retry loop never incremented retryCounter, so an operation that kept failing was retried forever instead of stopping at its retry limit.

  currentItemState = ItemProcessingState.INITIAL;
  requestToExecute = null;
  executionResult = null;
+ retryCounter++;
  assertInvariants(ItemProcessingState.INITIAL);
OpenSearch PR #11153 ↗
Accepted Python

Data loss · Airflow

session.expunge_all() detached every object from the shared database session, including rows the caller had queued but not yet committed, so they silently never got written.

  )
- session.expunge_all()
+ if conn:
+     session.expunge(conn)
  return conn
Airflow PR #63080 ↗
Accepted Go

Crash without a repository contract · go-git

Concurrent FindHash calls race on offsetHash and hit Go's fatal concurrent-map error.

- offsetHashIsFull bool
- mu               sync.RWMutex
+ offsetBuildOnce sync.Once
+ mu              sync.RWMutex
go-git PR #1729 ↗
Accepted C++

Crash without a repository contract · OpenCV

An out-of-range value indexes the color lookup table out of bounds, a memory-safety error in any C++ program.

  int tz = cz >> (lab_base_shift - lab_lut_shift);
+ CV_DbgCheck(tx, tx >= 0 && tx < LAB_LUT_DIM, "");
+ CV_DbgCheck(ty, ty >= 0 && ty < LAB_LUT_DIM, "");
+ CV_DbgCheck(tz, tz >= 0 && tz < LAB_LUT_DIM, "");
OpenCV PR #21257 ↗

Rejected candidates

A future issue or patch is not enough by itself: the expected behavior must have been knowable from the repository at the selected snapshot. These changes add features, improve performance, or otherwise do not repair repository-defined behavior.

Rejected Java

Not discoverable · Gson

Nothing in Gson's documentation promises that a JSON string is coerced when a Number is requested.

  case NUMBER:
+ case STRING:
    return new LazilyParsedNumber(
  default:
    throw new JsonSyntaxException(
Gson PR #964 ↗
Rejected Java

New feature · Quarkus

The change adds PATCH to the HttpMethod enum; the repository did not establish that support as existing behavior.

  enum HttpMethod {
      GET,
      HEAD,
+     PATCH,
      POST,
      PUT,
Quarkus PR #51729 ↗
Rejected Python

New feature · SymPy

The printer stated that Min and Max were unsupported, so adding printer support is a new feature rather than a bug fix.

  _known_functions = {
      'Abs': 'abs',
+     'Min': 'min',
+     'Max': 'max',
  }
SymPy PR #22914 ↗
Rejected Java

Claim versus code · OpenSearch

Although labeled as a bug fix, the change only adds a getKeepAlive() getter without repairing wrong behavior.

  public long getCreationTime() {
      return creationTime;
  }
+ public long getKeepAlive() {
+     return keepAlive;
+ }
OpenSearch PR #14495 ↗
Rejected Ruby

Not a correctness bug · Rails

Skipping a query for a contradictory WHERE clause improves performance, but the previous empty result was already correct.

+ if where_clause.contradiction?
+   return @async ? Promise::Complete.new({}) : {}
+ end
Rails PR #58058 ↗
Rejected JavaScript

New feature · React

Adding credentialless to the known boolean HTML attributes introduces support for a new attribute rather than fixing existing behavior.

  case 'autoPlay':
  case 'controls':
+ case 'credentialless':
  case 'default':
  case 'defer':
React PR #36148 ↗