SAP IDoc → canonical REST publishing layer

/* SAP MATMAS03 IDoc → Canonical Material REST payload mapping */

SAP_IDoc_Raw (MATMAS03)          →   Canonical_Rest_JSON
----------------------------------------------|--------------------------------
EDI_DC40-IDOCTYP                         →   data.type          = "Material"
EDI_DC40-MESCOD                          →   data.partitionCode
E1MARAM-MATNR                            →   data.materialNumber
E1MARAM-MEINS                            →   data.baseUnit.code
E1MARAM-MATKL                            →   data.materialGroup
E1MARCM-WERKS                            →   data.plants[].plantId
E1MARCM-LVORM                            →   data.plants[].isMarkedForDeletion
E1MAKTM-SPRAS                            →   data.descriptions[].language
E1MAKTM-MAKTX                            →   data.descriptions[].text

/* Transformation middleware (pseudo-Java): */

IdocSegment dc40   = idoc.getSegment("EDI_DC40");
IdocSegment maram  = idoc.getSegment("E1MARAM");
List<IdocSegment> plants = idoc.getSegments("E1MARCM");

CanonicalMaterial material = CanonicalMaterial.builder()
    .type("Material")
    .materialNumber(maram.getField("MATNR"))
    .baseUnit(Unit.of(maram.getField("MEINS")))
    .partitionCode(dc40.getField("MESCOD"))
    .build();

plants.forEach(p -> material.addPlant(
    p.getField("WERKS"),
    "X".equals(p.getField("LVORM"))
));

How we'd take this from discovery to a production-safe cutover — the phases, the canonical mapping, and the edge cases that bite.

This integration wraps SAP IDoc output in a canonical REST publishing layer, replacing fragile point-to-point IDoc parsing in downstream consumers. The pathway covers extraction from SAP ECC, transformation to a technology-agnostic canonical model, and exposure via a versioned REST API with event-driven notification.

1. Environment Discovery & IDoc Port Analysis — Identify the SAP Logical System names, Client, and RFC/IDoc ports (transaction WE21). Catalogue existing IDoc message types (e.g., ORDERS05, DESADV06, INVOIC03) and their active partner profiles (WE20). Confirm whether the middleware uses SAP PI/PO, SAP Integration Suite (Cloud Integration), or a third-party ESB (Boomi, MuleSoft, TIBCO). Pull a sample of 3–5 production IDocs per message type via WE02 for structural analysis.
2. Canonical Data Model Design — Define a JSON Schema canonical model that flattens IDoc segment hierarchies into resource-oriented objects. Separate core business events (OrderCreated, ShipmentConfirmed) from payload data. Align field names to business language (e.g., E1EDK01.BSTYP → documentType). Version the schema (e.g., 2.1.0) and publish to the enterprise API registry before coding begins.
3. IDoc Adapter Configuration — Configure the inbound IDoc adapter in the middleware. Set the Quality of Service to Exactly-Once if the ESB supports idempotent receivers; otherwise rely on SAP's guaranteed delivery with STAT status codes. Enable the ProcessingMode as Collect Messages if batching is needed. Map the SAP Message Type to the integration flow; set idocType and basicType parameters to match the profile from WE20.
4. Mapping & Transformation Build — Construct the message mapping between the IDoc XSD (generated from WE30) and the canonical JSON schema. Handle multi-occurrence segments (e.g., multiple E1EDP01 line items) by mapping to a JSON array. Convert SAP date formats (YYYYMMDD, YYYYMMDDHHMMSS) to ISO-8601. Translate EDIDC.PRNAM (processing program) and EDIDC.MESGF (message function) into canonical metadata fields. Add a sourceSystem: "SAP-ECC" and sourceId: "IDOC_NUM" envelope to every message.
5. REST Endpoint & Security — Deploy a POST endpoint (e.g., https://api.company.com/canonical/v2/events) with OAuth 2.0 client credentials. Configure rate limiting per consumer client ID. Enable request/response logging to a SIEM-compatible audit store (Splunk, ELK) without persisting sensitive payload fields. Set up a separate GET endpoint for event replay: GET /canonical/v2/events/{eventId}.
6. Event Notification Layer — Publish a lightweight event to a Kafka topic or webhook queue after successful transformation. The event payload carries only eventId, eventType, timestamp, and sourceId — never the full canonical payload — so consumers pull by reference. This decouples delivery from processing.
7. Error Handling & Dead-Letter Queue — Route mapping failures to a DLQ with original IDoc attachment. Configure alerting on EDIDS.STAMID status codes ≥ 28 (error). Implement a manual repair flow that resubmits from the DLQ after XSLT or mapping correction.
8. Regression & Load Testing — Simulate peak volumes (e.g., 500 IDocs/min during month-end) using a mock SAP system. Validate that no message loss occurs when the REST endpoint returns HTTP 503 and the middleware retries per its backoff policy. Confirm idempotency keys prevent duplicate canonical events on replay.
9. Cutover & Parallel Run — Shadow the existing IDoc-to-downstream feeds for 5 business days. Validate event counts in the canonical layer against SAP transaction BD87 reprocess counts. Flip consumer subscriptions one system at a time, with rollback to the legacy IDoc path if error rates exceed 0.1%.

Canonical Field Mapping

Edge Cases

• Empty mandatory segments — Some IDocs arrive without expected E1EDK01 or E1EDP01 blocks; the mapping must write null or skip the field rather than throw a null-pointer, or downstream validators will reject every message.
• Date field truncation in SAP — If SAP stores dates as DATS (8-char) and the source sends 2024010 (7-char) due to a leading-zero drop bug, the ISO-8601 conversion silently drops the day instead of failing loudly.
• Multi-byte characters in NAME1 — SAP codepage issues (e.g., 4103 vs UTF-8) corrupt downstream JSON when the REST layer serialises; require explicit codepage conversion at the IDoc adapter.
• Duplicate IDocs from BD87 reprocess — Manual reprocessing in SAP triggers the same IDoc number; without idempotency-key deduplication at the REST endpoint, consumers see duplicate OrderCreated events.
• Segment overflow (> 999 repetitions) — SAP limits segment repetitions; when E1EDP01 exceeds 999 lines the IDoc uses continuation segments (E1EDPT1); the mapping must reassemble the full hierarchy or silently truncate the payload.
• Partial reversal IDocs — ORDERS05 with function 004 (change) or 005 (cancel) arrive without a full replacement payload; the canonical event type must reflect the reversal, not the original document structure.

Cutover

Cutover must be fully reversible and reconciled. Before decommissioning any legacy IDoc consumer, export the last 30 days of EDIDC.DOCNUM ranges from SAP (transaction BD87) and compare against the canonical event log. A row-count match at the aggregate level, combined with sampling 5% of document IDs for field-level diff, constitutes a valid reconciliation gate. Keep the SAP IDoc port active for 90 days post-cutover as a shadow fallback; if the canonical layer reports error rates above the 0.1% threshold, redirect consumers back to the legacy feed without requiring a redeployment. Store the final mapping version number in the message envelope (mappingVersion: "2.1.0") so historical events can be replayed through the correct transformer if schema evolution breaks backward compatibility.