W-19424007 (SPIKE) Try Panel gRPC support with new proto definitions

[alexpmule]

Summary

• Introduced example.proto for gRPC service definition with SayHello method.
• Updated package.json and package-lock.json to include grpc-web and protobufjs dependencies.
• Enhanced ApiConsole to handle gRPC operations, including request schema extraction and dynamic message creation.
• Added styles for gRPC request panel in ApiConsoleStyles.js.
• Implemented form rendering for gRPC fields and handling of gRPC requests in the API console.

Spike: gRPC Integration in API Console

Objective

Evaluate the technical feasibility of integrating gRPC support into API Console, enabling visualization of gRPC API metadata (using JSON-LD generated by AMF) and testing operations (e.g., Greeter.SayHello1) via a "Try It" panel in the browser, while maintaining compatibility with REST/RAML.

Context

API Console currently supports REST/RAML and YAML APIs used by MuleSoft products like Design Center and Exchange. This spike explores:

1. Parsing gRPC models in JSON-LD format (e.g., grpc.json, with service Greeter, methods SayHello1/SayHello2, and messages HelloRequest/HelloReply).
2. Implementing a gRPC-Web client for browser-based requests.
3. Setting up a gRPC-Web proxy to connect the UI to gRPC servers.
4. Generating a consistent UI for navigation and testing, based on JSON-LD.

Proposed Architecture

Main Components

graph TD
    A[API Console UI] -->|Parses| B[AMF JSON-LD Parser]
    A -->|Generates| C[gRPC Try Panel]
    C -->|Sends request| D[gRPC-Web Client]
    D -->|Translates HTTP/1.1 to HTTP/2| E[gRPC-Web Proxy]
    E -->|Native gRPC| F[gRPC Server]
    B -->|Extracts| G[JSON-LD Model: grpc.json]
    G -->|Converts to| H[Proto Schema]
    H -->|Used by| D

Loading

Technology Stack

• Frontend: LitElement (API Console base).
• gRPC Client: grpc-web (for browser).
• Schema Parsing: protobufjs (to interpret JSON-LD and build messages).
• Proxy: @grpc-web/proxy (Node.js) or Envoy (production).
• Server: Node.js with @grpc/grpc-js (for local testing).
• Dependencies: npm install grpc-web protobufjs @grpc-web/proxy @grpc/grpc-js.

Implementation

1. Detecting gRPC Operations

Detect if an operation is gRPC by checking the mediaType in the JSON-LD:

_isGrpcOperation(selected, webApi) {
  const method = this._computeMethodModel(webApi, selected);
  const request = this._computePayload(method);
  const mediaType = this._getValue(request, this.ns.aml.vocabularies.core.mediaType);
  return mediaType === 'application/grpc';
}

2. Parsing JSON-LD for Schemas

Extract services, methods, and messages from JSON-LD to generate the UI:

_getGrpcRequestSchema(selected, webApi) {
  const method = this._computeMethodModel(webApi, selected);
  const request = this._computePayload(method);
  const schema = this._getValue(request, this.ns.aml.vocabularies.shapes.schema);
  const properties = this._getValueArray(schema, 'property');
  
  return {
    name: this._getValue(schema, this.ns.aml.vocabularies.core.name), // e.g., "HelloRequest"
    fields: properties.map(prop => ({
      name: this._getValue(prop, this.ns.aml.vocabularies.core.name), // e.g., "name"
      type: this._getValue(prop, this.ns.aml.vocabularies.shapes.range), // e.g., string
      required: this._getValue(prop, this.ns.aml.vocabularies.shapes.minCount) > 0
    }))
  };
}

3. UI for the Try Panel

Add a panel with a form and a "Try gRPC Request" button:

html`
  <div class="grpc-request-panel">
    <h3>gRPC Request: ${methodName}</h3>
    <div class="grpc-form">
      ${schema.fields.map(field => this._renderGrpcField(field))}
    </div>
    <anypoint-button @click="${this._handleGrpcRequest}">
      Try gRPC Request
    </anypoint-button>
  </div>
`;

4. gRPC-Web Client

Use grpc-web to send requests to the proxy:

import { GrpcWebClientBase } from 'grpc-web';

async _handleGrpcRequest() {
  const client = new GrpcWebClientBase({
    format: 'text', // grpcwebtext for browser compatibility
    suppressCorsPreflight: true
  });
  const message = this._buildProtobufMessage(this.formData); // Convert form to protobuf
  try {
    const response = await client.rpcCall(
      'http://localhost:8080/helloworld.Greeter/SayHello1',
      message,
      {}, // Metadata
      null, // Call options
      () => {}
    );
    this._displayResponse(response); // Display in UI
  } catch (err) {
    this._displayError(err);
  }
}

5. gRPC-Web Proxy Configuration

The proxy is critical to translate gRPC-Web to native gRPC, as browsers do not support full HTTP/2. We use @grpc-web/proxy for local testing:

const express = require('express');
const cors = require('cors');
const { createServer } = require('@grpc-web/proxy');

const app = express();
app.use(cors());
const proxy = createServer({
  upstream: 'http://localhost:50051', // gRPC server
  protoPath: './simple.proto', // Based on grpc.json
  binary: true
});
app.use('/', proxy);
app.listen(8080, () => console.log('gRPC-Web proxy on :8080'));

6. Local gRPC Server (for Testing)

Implemented a test server in Node.js to simulate Greeter:

const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const packageDefinition = protoLoader.loadSync('simple.proto');
const helloworld = grpc.loadPackageDefinition(packageDefinition).helloworld;

function sayHello(call, callback) {
  const name = call.request.getName();
  callback(null, { message: `Hello ${name}! Wadus received.` });
}

const server = new grpc.Server();
server.addService(helloworld.Greeter.service, { SayHello1: sayHello, SayHello2: sayHello });
server.bindAsync('0.0.0.0:50051', grpc.ServerCredentials.createInsecure(), () => {
  server.start();
  console.log('gRPC server on :50051');
});

Challenges and Considerations

1. Proxy Configuration

• Need: Browsers do not support native gRPC (HTTP/2 with trailers). A proxy (@grpc-web/proxy or Envoy) translates gRPC-Web to native gRPC.
• Adoption by Other Teams:
  • Installable Library: @grpc-web/proxy is an npm module, easy to share via an internal registry (e.g., @mulesoft/grpc-web-proxy).
  • Alternative: Envoy for production (configured via envoy.yaml, run with Docker).
  • Recommendation: Package the proxy as an npm module with a simple CLI (e.g., setupGrpcWebProxy --port 8080 --upstream :50051).

2. JSON-LD Parsing

• Challenge: The JSON-LD (grpc.json) contains services (Greeter), methods (SayHello1), and messages (HelloRequest with name, wadus). Parsing shacl#property and shapes#range recursively for nested messages and enums.
• Solution: Use protobufjs to convert JSON-LD to a dynamic protobuf schema, generating forms in the try panel.

3. RPC Types

• Unary: Implemented (e.g., SayHello1).
• Streaming: Pending; requires UI for handling streams (e.g., "send chunk" buttons, real-time logs).
• Challenge: Differentiate in the UI (icons for unary vs. streaming) and handle grpc-web callbacks.

4. Security

• CORS: Configured in the proxy (cors middleware).
• TLS/SSL: Insecure for local testing; production requires certificates.
• Authentication: Support for metadata (e.g., OAuth tokens) in grpc-web.

5. MuleSoft Compatibility

• Integration with Design Center/Exchange: Extend the AMF parser to map gRPC to RAML-like structures, enabling previews in Design Center.
• Challenge: Align with MuleSoft's asset model, treating gRPC services as "endpoints".

Next Steps

Phase 1: Fundamentals

1. Implement a robust JSON-LD to protobuf parser (handle oneof, enums).
2. Support unary calls in the try panel.
3. Document proxy setup (@grpc-web/proxy) for teams.

Phase 2: Streaming and Features

1. Support streaming (client/server/bidirectional) in the UI.
2. Add metadata and headers in the try panel.
3. Implement cancellation and timeouts.

Phase 3: UX and Adoption

1. Dedicated panel for responses and errors.
2. Autocomplete and form validation based on JSON-LD.
3. Publish @mulesoft/grpc-web-proxy as an internal npm module.
4. Guide for integration with Design Center/Exchange.

Conclusions

Feasibility

gRPC integration in API Console is feasible using:

• JSON-LD for metadata (parsed with protobufjs).
• grpc-web for browser requests.
• @grpc-web/proxy or Envoy to connect to gRPC servers.
• LitElement's modular architecture for REST compatibility.

Recommendations

1. Prioritize unary calls for an MVP.
2. Standardize the proxy as an npm module for adoption.
3. Align with MuleSoft products via AMF mappings.
4. Use tools like grpcui for initial local testing.

Risks

1. Proxy Complexity: Setup may be a barrier for teams unfamiliar with gRPC.
2. Streaming UX: Requires significant try panel changes.
3. Maintenance: Support for new gRPC-Web and AMF versions.
4. Performance: Parsing large JSON-LD and protobuf encoding in the browser.

References

• gRPC-Web GitHub
• Protobuf.js Documentation
• AMF Documentation
• Envoy Proxy Docs
• gRPCui Tool