Skip to content

Fix ckan export - #111

Merged
karlbrink merged 17 commits into
masterfrom
fix_ckan_export
Aug 21, 2026
Merged

Fix ckan export#111
karlbrink merged 17 commits into
masterfrom
fix_ckan_export

Conversation

@karlbrink

Copy link
Copy Markdown

No description provided.

karlbrink and others added 7 commits July 21, 2026 19:03
…ve geometries

This change hardens the Linked Data Proxy and GML parsing pipeline to work reliably
with stricter and non-uniform WFS implementations (especially QGIS/GeoServer-like
WFS 1.1.0 services), while keeping behavior generic across services.

Key improvements:

Added robust geometry parsing in GML 3 factory:
  namespace-tolerant/local-name based geometry detection
  support for Curve and MultiCurve variants
  support for embedded LineStringSegment paths in MultiCurve structures
  improved fallback geometry discovery when geometry elements are nested or prefixed differently
Improved feature serialization:
  output GeoJSON geometry as null (instead of empty string) when no geometry is available
  keeps GeoJSON output standards-compliant and avoids downstream parsing issues
Fixed WFS request construction for strict servers:
  version-specific request parameters:
    WFS 2.x uses TYPENAMES + COUNT
    WFS 1.1.0 uses TYPENAME + MAXFEATURES
  normalized request key casing (SERVICE/REQUEST/VERSION)
  removed duplicate max-features parameter in paging GET requests
Added request-version fallback strategy:
  detect service version and retry with compatible versions when needed
  improved handling of WFS exception responses before deciding fallback
Improved proxy paging/filter compatibility:
  support STARTINDEX as alias parameter
  convert FES filter namespace/elements to OGC equivalents for non-2.x services
Reworked GeoJSON bbox handling:
  generic recursive bbox calculation for Point/Line/Polygon/Multi*/GeometryCollection
  consistent feature-collection bbox accumulation
  reduced reliance on brittle geometry-type-specific bbox code
Hardened cached count behavior:
  detect implausible cached counts
  trigger recount and refresh cache when cached values are stale/invalid
Prevented accidental feature loss in conversion paths:
  do not drop features simply because geometry is null
  guard geometry-dependent logic (bbox/type extraction) accordingly
Result:
The proxy now handles a broader range of real-world WFS services more reliably,
including WFS 1.1.0 endpoints and MultiCurve-based datasets, with improved stability
for paging, filtering, geometry conversion, and bbox generation.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Added additional SQL query to fetch German category codes.
@github-actions

Copy link
Copy Markdown

Code Review by Gemini

Here are the bugs and issues identified in the pull request along with suggested fixes.


1. Swapped X/Y Coordinates in Bounding Box Calculation

File: http/php/mod_linkedDataProxy.php
Lines: 308–313, 325–328

In calculateBboxFromGeojsonGeometry(), $minLat / $maxLat represent Y coordinates (latitude) and $minLon / $maxLon represent X coordinates (longitude). When assigning properties to the $bbox object (and when comparing geometry collection parts), minx was assigned $minLat and miny was assigned $minLon. This swaps X and Y coordinates.

// http/php/mod_linkedDataProxy.php

// Lines 308-313
			if ($bboxPart !== false) {
				if ($minLon > $bboxPart->minx) { $minLon = $bboxPart->minx; }
				if ($minLat > $bboxPart->miny) { $minLat = $bboxPart->miny; }
				if ($maxLon < $bboxPart->maxx) { $maxLon = $bboxPart->maxx; }
				if ($maxLat < $bboxPart->maxy) { $maxLat = $bboxPart->maxy; }
			}

// Lines 325-328
	$bbox = new stdClass();
	$bbox->minx = $minLon;
	$bbox->miny = $minLat;
	$bbox->maxx = $maxLon;
	$bbox->maxy = $maxLat;

2. Relative Path Used in file_get_contents

File: http/plugins/mb_downloadFeedServer.php
Line: 11

file_exists() uses dirname(__FILE__) for an absolute path, but file_get_contents() uses a relative path ../../conf/excludeFromAtomFeedClient.json. If the current working directory differs from the script directory, file_get_contents() will fail.

// http/plugins/mb_downloadFeedServer.php line 11
	if (file_exists ( dirname ( __FILE__ ) . "/../../conf/excludeFromAtomFeedClient.json" )) {
	    $configObject = json_decode ( file_get_contents ( dirname ( __FILE__ ) . "/../../conf/excludeFromAtomFeedClient.json" ) );
	}

3. Typo in Standard Name

File: http/php/mod_exportMapbenderMetadata2Ckan.php
Line: 1089

There is a typo in the ISO standard description string (ISO19168-1:20202). The standard version year contains an extra 2.

// http/php/mod_exportMapbenderMetadata2Ckan.php line 1089
"description" =>   "Objektart: " . $value1->resourceName. " - ISO19168-1:2020 API",

4. Docblock Parameter Mismatch

File: http/classes/class_metadata.php
Lines: 2360–2367

The PHPDoc docblock for computeIsHVD() documents $inspireCats as a second parameter (@param array $inspireCats), but the function signature only accepts $customCats. Update the docblock to match the method signature.

// http/classes/class_metadata.php lines 2360-2367
	/**
	 * Computes whether a resource is a High-Value Dataset (HVD)
	 * based on the hvd_cats.json configuration.
	 * Checks if any of the resource's custom category IDs
	 * match those configured as HVD categories.
	 *
	 * @param array $customCats Array of custom category IDs (strings)
	 * @return bool true if the resource qualifies as HVD
	 */
	private function computeIsHVD($customCats) {

@github-actions

Copy link
Copy Markdown

Code Review by Gemini

Here is a code review highlighting issues and suggested improvements found in the pull request:


1. http/delete.me

Issue: Temporary / debug file included in the pull request.

  • Line 1:
    this is my testfile for ai checks
    
  • Recommendation: Remove http/delete.me before merging.

2. http/php/mod_exportMapbenderMetadata2Ckan.php

Issue: Duplicate case "wfsrequest": label inside the switch ($value1->type) block. In PHP, duplicate switch cases result in dead code because execution will always jump to the first matching case, rendering the second wfsrequest block unreachable.

  • Lines to change (http/php/mod_exportMapbenderMetadata2Ckan.php):
    case "directwfs":
        $featuretypeAccessResource_2 = array("name" => "WFS-Schnittstelle",
            "description" =>   "Objektart: " . $value1->resourceName. " - WFS",
            "format" => "WFS",
            "url" => str_replace($mapbenderWebserviceUrl, $mapbenderBaseUrl, $value1->accessClient),
            "original_url" => str_replace("http://127.0.0.1", $baseUrlPortal, str_replace("http://localhost", $baseUrlPortal, $value1->originalGetCapabilitiesUrl)),
            "id" => $gpDataset->uuid . "_wfs_interface_" . str_replace(":", "__colon__", $value1->resourceName) . "_" . $value1->serviceId,
            "license_id" => $inspireAtomFeedsLicenseId,
            "license_source_note" => $value1->licenseSourceNote
        );
        $resourceArray[] = $featuretypeAccessResource_2;
        break;

3. http/classes/class_metadata.php

Issue 1: In computeIsHVD(), casting non-numeric or empty string values like "" using (int)$catId evaluates to 0. If 0 exists in $this->hvdCustomCats, this yields false positives.

  • Lines to change (http/classes/class_metadata.php):
    private function computeIsHVD($customCats) {
        if (!empty($this->hvdCustomCats) && is_array($customCats)) {
            foreach ($customCats as $catId) {
                if (is_numeric($catId) && in_array((int)$catId, $this->hvdCustomCats, true)) {
                    return true;
                }
            }
        }
        return false;
    }

Issue 2: The DocBlock comment for computeIsHVD() documents $inspireCats as a second parameter, but the method signature accepts only one argument ($customCats). Update the PHPDoc comment to match the function signature.


4. http/plugins/mb_downloadFeedServer.php

Issue: Potential file_get_contents() failure due to relative path usage. While file_exists() uses dirname(__FILE__) . "/../../conf/excludeFromAtomFeedClient.json", file_get_contents() uses ../../conf/excludeFromAtomFeedClient.json". If the script is executed from a different working directory, file_get_contents() will fail.

  • Lines to change (http/plugins/mb_downloadFeedServer.php):
    if (file_exists ( dirname ( __FILE__ ) . "/../../conf/excludeFromAtomFeedClient.json" )) {
        $configObject = json_decode ( file_get_contents ( dirname ( __FILE__ ) . "/../../conf/excludeFromAtomFeedClient.json" ) );
    }

@github-actions

Copy link
Copy Markdown

ERROR: # Code Review by Gemini
An unexpected error occurred in Client call: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}

@github-actions

Copy link
Copy Markdown

Code Review by Gemini

Here is the review of the changes in this pull request along with identified bugs and suggested improvements:


1. Leftover Test File

http/delete.me

A test file http/delete.me was added in this commit. Test/temporary trigger files should be removed before merging into production.

Action Required: Remove http/delete.me.


2. Inverted Bounding Box Coordinates and Coordinate System Boundary Handling

http/php/mod_linkedDataProxy.php

In calculateBboxFromGeojsonGeometry:

function calculateBboxFromGeojsonGeometry($geometry) {
	if (!is_object($geometry) || !isset($geometry->type)) {
		return false;
	}
	$minLat = 90;
	$minLon = 180;
	$maxLat = -90;
	$maxLon = -180;
...
	$bbox = new stdClass();
	$bbox->minx = $minLat;
	$bbox->miny = $minLon;
	$bbox->maxx = $maxLat;
	$bbox->maxy = $maxLon;
	return $bbox;
}

Issues:

  1. Inverted Coordinates: minx and maxx are assigned $minLat/$maxLat (Y coordinates/latitude), while miny and maxy are assigned $minLon/$maxLon (X coordinates/longitude). This swaps the X and Y axes in generated bounding box outputs.
  2. Hardcoded Degree Range: Initializing bounds to 90/-90 and 180/-180 fails for non-geographic projected coordinate systems (e.g. UTM / EPSG:25832 meter-based coordinates).

Suggested Change:
In http/php/mod_linkedDataProxy.php, update the initialization, assignment, and properties:

function calculateBboxFromGeojsonGeometry($geometry) {
	if (!is_object($geometry) || !isset($geometry->type)) {
		return false;
	}
	$minLat = INF;
	$minLon = INF;
	$maxLat = -INF;
	$maxLon = -INF;

	if ($geometry->type === "GeometryCollection" && isset($geometry->geometries) && is_array($geometry->geometries)) {
		foreach ($geometry->geometries as $geometryPart) {
			$bboxPart = calculateBboxFromGeojsonGeometry($geometryPart);
			if ($bboxPart !== false) {
				if ($minLon > $bboxPart->minx) { $minLon = $bboxPart->minx; }
				if ($minLat > $bboxPart->miny) { $minLat = $bboxPart->miny; }
				if ($maxLon < $bboxPart->maxx) { $maxLon = $bboxPart->maxx; }
				if ($maxLat < $bboxPart->maxy) { $maxLat = $bboxPart->maxy; }
			}
		}
	}
	else if (isset($geometry->coordinates)) {
		$coords = json_decode(json_encode($geometry->coordinates), true);
		updateBboxAccumulatorFromCoordinates($coords, $minLat, $minLon, $maxLat, $maxLon);
	}

	if ($minLat === INF || $minLon === INF || $maxLat === -INF || $maxLon === -INF) {
		return false;
	}

	$bbox = new stdClass();
	$bbox->minx = $minLon;
	$bbox->miny = $minLat;
	$bbox->maxx = $maxLon;
	$bbox->maxy = $maxLat;
	return $bbox;
}

3. Partial String Matching in Subdomain/Path for Whitelist Checks

http/plugins/mb_downloadFeedServer.php

function isWhitelistedDownloadFeedUrl($serviceFeedUrl) {
	global $urlsWhitelist;
	if ($urlsWhitelist === false || !is_array($urlsWhitelist) || count($urlsWhitelist) === 0) {
		return false;
	}
	$parsedUrl = parse_url($serviceFeedUrl);
	$host = isset($parsedUrl['host']) ? strtolower($parsedUrl['host']) : '';
	foreach ($urlsWhitelist as $whitelistEntry) {
		$whitelistEntry = strtolower(trim($whitelistEntry));
		if ($whitelistEntry === '') {
			continue;
		}
		if ($host !== '' && $host === $whitelistEntry) {
			return true;
		}
		if (strpos(strtolower($serviceFeedUrl), $whitelistEntry) !== false) {
			return true;
		}
	}
	return false;
}

Issue:
The strpos(strtolower($serviceFeedUrl), $whitelistEntry) !== false check matches any occurrences of the whitelist string anywhere inside the URL (such as path or query parameters). This can cause arbitrary non-whitelisted domains to bypass blacklist validation.

Suggested Change:
In http/plugins/mb_downloadFeedServer.php, replace the matching loop:

function isWhitelistedDownloadFeedUrl($serviceFeedUrl) {
	global $urlsWhitelist;
	if ($urlsWhitelist === false || !is_array($urlsWhitelist) || count($urlsWhitelist) === 0) {
		return false;
	}
	$parsedUrl = parse_url($serviceFeedUrl);
	$host = isset($parsedUrl['host']) ? strtolower($parsedUrl['host']) : '';
	if ($host === '') {
		return false;
	}
	foreach ($urlsWhitelist as $whitelistEntry) {
		$whitelistEntry = strtolower(trim($whitelistEntry));
		if ($whitelistEntry === '') {
			continue;
		}
		if ($host === $whitelistEntry || sprintf('.%s', $host) === strstr($host, sprintf('.%s', $whitelistEntry))) {
			return true;
		}
	}
	return false;
}

4. Whitespace Parsing in GML Coordinate Delimiters

http/classes/class_gml_3_factory.php

$coordArray = explode(' ', trim($coordsDom->nodeValue));

Issue:
GML coordinate blocks frequently contain multi-space separations, newlines, or tabs. Using explode(' ', ...) creates empty array elements which leads to invalid coordinate indices during iteration.

Suggested Change:
In http/classes/class_gml_3_factory.php, update coordinate parsing in parseCurve (and existing methods using explode(' ', ...)):

Lines 198, 226, 290, etc.:

- $coordArray = explode(' ', trim($coordsDom->nodeValue));
+ $coordArray = preg_split('/\s+/', trim($coordsDom->nodeValue));

5. Docblock Signature Mismatch

http/classes/class_metadata.php

	/**
	 * Computes whether a resource is a High-Value Dataset (HVD)
	 * based on the hvd_cats.json configuration.
	 * Checks if any of the resource's custom or inspire category IDs
	 * match those configured as HVD categories.
	 *
	 * @param array $customCats Array of custom category IDs (strings)
	 * @param array $inspireCats Array of inspire category IDs (strings)
	 * @return bool true if the resource qualifies as HVD
	 */
	private function computeIsHVD($customCats) {

Issue:
The docblock specifies $inspireCats as a second parameter, but computeIsHVD accepts only $customCats.

Suggested Change:
In http/classes/class_metadata.php, update the PHPDoc block:

	/**
	 * Computes whether a resource is a High-Value Dataset (HVD)
	 * based on the hvd_cats.json configuration.
	 *
	 * @param array $customCats Array of custom category IDs (strings)
	 * @return bool true if the resource qualifies as HVD
	 */
	private function computeIsHVD($customCats) {

@karlbrink
karlbrink marked this pull request as ready for review August 21, 2026 12:27
Copilot AI lite review requested due to automatic review settings August 21, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves CKAN export output and hardens/extends several Mapbender HTTP endpoints involved in WFS/linked-data access and INSPIRE download feed handling.

Changes:

  • Add whitelist support to the download feed server’s URL filtering and expand the default blacklist distribution file.
  • Improve mod_linkedDataProxy.php robustness: WFS version fallback chain, better bbox handling for more GeoJSON geometry types, and STARTINDEX compatibility.
  • Extend CKAN metadata export handling for additional AtomFeed-linked resource types; add metadata search facets and HVD logic updates.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
http/plugins/mb_downloadFeedServer.php Adds whitelist logic to bypass blacklist for trusted feed URLs (needs tightening + path fix).
http/php/mod_linkedDataProxy.php Adds WFS paging/version fallbacks and generalized GeoJSON bbox computation.
http/php/mod_exportMapbenderMetadata2Ckan.php Exports additional AtomFeed-linked resource types into CKAN distributions.
http/delete.me New test file added (should be removed).
http/classes/class_wfs.php Adjusts WFS KVP parameter casing / typeName parameter handling and paging request building.
http/classes/class_metadata.php Adds adminTypes facet/filtering, improves HVD filtering logic, and enriches metadata JSON fields.
http/classes/class_gml_feature.php Outputs null for missing geometry in GeoJSON.
http/classes/class_gml_3_factory.php Improves GML3 geometry parsing (Curve + namespace-agnostic XPath), geometry detection, and SRS handling.
conf/excludeFromAtomFeedClient.json-dist Expands blacklist patterns and adds a whitelist example section.
.github/workflows/gemini-review.yml Changes workflow trigger to run only when the gemini-review label is added.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread http/plugins/mb_downloadFeedServer.php Outdated
Comment thread http/classes/class_metadata.php Outdated
Comment thread http/plugins/mb_downloadFeedServer.php
Comment thread http/classes/class_gml_3_factory.php
Comment thread http/classes/class_metadata.php
karlbrink and others added 5 commits August 21, 2026 14:34
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@karlbrink
karlbrink merged commit 7896820 into master Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants