FA-69986 / Map projection transforms / Open access

Web mercator ground resolution and scale denominator: rounding stage · case 01

Scale denominators at high zoom levels are off by several hundred.

Verified by executionVariant 1 · 8 checks per implementationDownload source bundle ↓JSON ↗

ROOT CAUSE

The resolution is rounded to 2 decimals before the scale is derived, amplifying the rounding error by dpi/0.0254.

THE FAILURE

The resolution is rounded to 2 decimals before the scale is derived, amplifying the rounding error by dpi/0.0254.

Unsuccessful approach: Rounding to 4 decimals shrinks but does not remove the error, which is still visible after multiplying by ~3780.

Case contract

Input [lat, z, tile_size, dpi]. Ground resolution in metres per pixel is 2*pi*6378137*cos(lat)/(tile_size*2**z) with the equatorial circumference computed exactly. The map scale denominator is resolution*dpi/0.0254. Return [resolution rounded to 6 decimals, scale denominator rounded to the nearest integer] computed from the unrounded resolution.

Why this case matters

Scale bars, level-of-detail choice and print exports depend on these values.

1 / The failure

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(x):
    lat, z, tile, dpi = x
    circumference = 2 * math.pi * 6378137.0
    res = circumference * math.cos(math.radians(lat)) / (tile * 2 ** z)
    scale = round(res, 2) * dpi / 0.0254
    return [round(res, 6), round(scale)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('control #0', [-0.88, 17, 256, 72], [1.194188, 3385]), ('control #1', [-33.07, 0, 512, 72], [65591.886576, 185929757]), ('control #2', [-22.5, 12, 512, 72], [17.654651, 50045]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #4', [30.21, 20, 256, 72], [0.129015, 366]), ('control #5', [-75.78, 4, 256, 300], [2403.383393, 28386418]), ('control #6', [-15.49, 9, 512, 72], [147.321225, 417603]), ('control #7', [76.6, 19, 512, 300], [0.034598, 409])], [('control #1', [-33.07, 0, 512, 72], [65591.886576, 185929757]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #5', [-75.78, 4, 256, 300], [2403.383393, 28386418]), ('control #6', [-15.49, 9, 512, 72], [147.321225, 417603]), ('control #7', [76.6, 19, 512, 300], [0.034598, 409]), ('control #8', [20.0, 19, 512, 72], [0.140288, 398]), ('control #9', [-14.77, 4, 256, 96], [9460.648974, 35756784]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053])], [('control #2', [-22.5, 12, 512, 72], [17.654651, 50045]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #8', [20.0, 19, 512, 72], [0.140288, 398]), ('control #9', [-14.77, 4, 256, 96], [9460.648974, 35756784]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053]), ('control #11', [-18.37, 9, 256, 72], [290.167542, 822522]), ('control #12', [-51.53, 14, 512, 300], [2.97199, 35102]), ('control #13', [-69.8, 6, 256, 300], [844.594183, 9975522])], [('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053]), ('control #11', [-18.37, 9, 256, 72], [290.167542, 822522]), ('control #12', [-51.53, 14, 512, 300], [2.97199, 35102]), ('control #13', [-69.8, 6, 256, 300], [844.594183, 9975522]), ('control #14', [-23.41, 10, 256, 72], [140.290274, 397673]), ('control #15', [-37.12, 12, 256, 96], [30.474424, 115179]), ('boundary #16', [0.0, 0, 256, 96], [156543.033928, 591658711])], [('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #4', [30.21, 20, 256, 72], [0.129015, 366]), ('control #14', [-23.41, 10, 256, 72], [140.290274, 397673]), ('control #15', [-37.12, 12, 256, 96], [30.474424, 115179]), ('boundary #16', [0.0, 0, 256, 96], [156543.033928, 591658711]), ('boundary #17', [60.0, 1, 512, 96], [19567.879241, 73957339]), ('regression #18', [45.0, 10, 512, 90.7], [54.049141, 193002]), ('boundary #19', [-60.0, 3, 256, 300], [9783.939621, 115558342])]]
for label, args, expected in fixtures[N-1]:
    check(label, solve(args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
control #0	`[1.194188, 3373]`	`[1.194188, 3385]`	Failed
control #1	`[65591.886576, 185929767]`	`[65591.886576, 185929757]`	Failed
control #2	`[17.654651, 50031]`	`[17.654651, 50045]`	Failed
control #3	`[37.345917, 441142]`	`[37.345917, 441094]`	Failed
control #4	`[0.129015, 369]`	`[0.129015, 366]`	Failed
control #5	`[2403.383393, 28386378]`	`[2403.383393, 28386418]`	Failed
control #6	`[147.321225, 417600]`	`[147.321225, 417603]`	Failed
control #7	`[0.034598, 354]`	`[0.034598, 409]`	Failed

SHA-256 / 56569c87e9e41ce4730eb7e5b95e44ffb93f71f94ab6d49d1bef6eba56eab246

2 / The unsuccessful fix

Exit 1

"""Failure Map reference implementation. Python standard library only."""
import json
import math
N = 1
observations = []
def solve(x):
    lat, z, tile, dpi = x
    circumference = 2 * math.pi * 6378137.0
    res = circumference * math.cos(math.radians(lat)) / (tile * 2 ** z)
    scale = round(res, 4) * dpi / 0.0254
    return [round(res, 6), round(scale)]
def check(label, actual, expected):
    observations.append({"check": label, "actual": actual, "expected": expected, "passed": actual == expected})
fixtures = [[('control #0', [-0.88, 17, 256, 72], [1.194188, 3385]), ('control #1', [-33.07, 0, 512, 72], [65591.886576, 185929757]), ('control #2', [-22.5, 12, 512, 72], [17.654651, 50045]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #4', [30.21, 20, 256, 72], [0.129015, 366]), ('control #5', [-75.78, 4, 256, 300], [2403.383393, 28386418]), ('control #6', [-15.49, 9, 512, 72], [147.321225, 417603]), ('control #7', [76.6, 19, 512, 300], [0.034598, 409])], [('control #1', [-33.07, 0, 512, 72], [65591.886576, 185929757]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #5', [-75.78, 4, 256, 300], [2403.383393, 28386418]), ('control #6', [-15.49, 9, 512, 72], [147.321225, 417603]), ('control #7', [76.6, 19, 512, 300], [0.034598, 409]), ('control #8', [20.0, 19, 512, 72], [0.140288, 398]), ('control #9', [-14.77, 4, 256, 96], [9460.648974, 35756784]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053])], [('control #2', [-22.5, 12, 512, 72], [17.654651, 50045]), ('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #8', [20.0, 19, 512, 72], [0.140288, 398]), ('control #9', [-14.77, 4, 256, 96], [9460.648974, 35756784]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053]), ('control #11', [-18.37, 9, 256, 72], [290.167542, 822522]), ('control #12', [-51.53, 14, 512, 300], [2.97199, 35102]), ('control #13', [-69.8, 6, 256, 300], [844.594183, 9975522])], [('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #10', [6.58, 15, 256, 300], [4.745845, 56053]), ('control #11', [-18.37, 9, 256, 72], [290.167542, 822522]), ('control #12', [-51.53, 14, 512, 300], [2.97199, 35102]), ('control #13', [-69.8, 6, 256, 300], [844.594183, 9975522]), ('control #14', [-23.41, 10, 256, 72], [140.290274, 397673]), ('control #15', [-37.12, 12, 256, 96], [30.474424, 115179]), ('boundary #16', [0.0, 0, 256, 96], [156543.033928, 591658711])], [('control #3', [75.86, 10, 256, 300], [37.345917, 441094]), ('control #4', [30.21, 20, 256, 72], [0.129015, 366]), ('control #14', [-23.41, 10, 256, 72], [140.290274, 397673]), ('control #15', [-37.12, 12, 256, 96], [30.474424, 115179]), ('boundary #16', [0.0, 0, 256, 96], [156543.033928, 591658711]), ('boundary #17', [60.0, 1, 512, 96], [19567.879241, 73957339]), ('regression #18', [45.0, 10, 512, 90.7], [54.049141, 193002]), ('boundary #19', [-60.0, 3, 256, 300], [9783.939621, 115558342])]]
for label, args, expected in fixtures[N-1]:
    check(label, solve(args), expected)
print(json.dumps({"observations": observations, "passed": all(x["passed"] for x in observations)}, ensure_ascii=False))
raise SystemExit(0 if all(x["passed"] for x in observations) else 1)

Boundary fixture	Actual	Expected	Outcome
control #0	`[1.194188, 3385]`	`[1.194188, 3385]`	Passed
control #1	`[65591.886576, 185929757]`	`[65591.886576, 185929757]`	Passed
control #2	`[17.654651, 50045]`	`[17.654651, 50045]`	Passed
control #3	`[37.345917, 441093]`	`[37.345917, 441094]`	Failed
control #4	`[0.129015, 366]`	`[0.129015, 366]`	Passed
control #5	`[2403.383393, 28386418]`	`[2403.383393, 28386418]`	Passed
control #6	`[147.321225, 417603]`	`[147.321225, 417603]`	Passed
control #7	`[0.034598, 409]`	`[0.034598, 409]`	Passed

SHA-256 / b7cef3ff480375da54a5006bdfd13ffd071130e61d9eaf41a7bfe06cf8f7f71f

HELD IN THE MEMBER ARCHIVE

The verified repair and its recorded checks are member-only.

This mechanism has 8 recorded checks per implementation. The open-access tier publishes the failure and the unsuccessful fix; the repaired source that passes every check, and the observations that prove it, are available to members.

Every case sharing this mechanism uses the same contract and the same repair, so this one record is held back for all of them.

Member access is invitation-based. Sign in with your invited account to inspect the repair.

Verification & scope

Stipulated deterministic toy contract on a bounded input domain; results are rounded as stated and no conformance with any published standard or library is claimed. This reproducer isolates one failure mechanism. Results cover the supplied fixtures. Variants within a family share a test contract and should remain grouped when constructing evaluation splits. Related mechanisms with a shared evaluation_group must also remain together; these controlled models are not independent production incidents.

Observations recorded using Python 3.12.14 at 2026-09-29T14:48:16.452259+00:00.

Case digest / 1ba3ffdb89bf456eade1f801c84ce0210e79c372e27de40a8b58e24e3afc84a1