DSL Build-Time Pre-Compilation (OAL/MAL/LAL/Hierarchy)

Context

SkyWalking OAP uses four DSL compilers that generate Java bytecode at startup via Javassist:

All four share the same pipeline (since upstream PR #13723):

DSL text → ANTLR4 parse → Immutable AST → Javassist bytecode → .class

GraalVM native image cannot run Javassist at runtime. Solution: run all four compilers at build time via their exposed APIs (setClassOutputDir(), setClassNameHint()), capture .class files into the precompiler output JAR. At runtime, load pre-compiled classes instead of compiling.

Build-Time Compilation

Module: build-tools/precompiler

The precompiler invokes each v2 compiler at build time, capturing generated .class files to the output directory. All classes are packaged into precompiler-*-generated.jar, which is on the native-image build classpath.

OAL

Uses OALEngineV2.start() with setOpenEngineDebug(true) and setGeneratedFilePath() to export classes. Processes all 9 OAL defines:

Generated class packages:

• Metrics: org.apache.skywalking.oap.server.core.source.oal.rt.metrics.*
• Builders: org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.*
• Dispatchers: org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.*

MAL

Uses MALClassGenerator with setClassOutputDir() and setClassNameHint() for deterministic class naming.

For each MAL YAML rule file (71 files, ~1250 rules):
  1. Parse YAML → List<MetricsRule>
  2. For each rule:
     a. generator.setClassNameHint(sanitize(yamlSourceId + "_" + metricName))
     b. generator.setYamlSource(yamlSourceId)
     c. MalExpression malExpr = generator.compile(metricName, expression)
     d. .class file auto-written to classOutputDir
     e. Record mapping: sha256(expression)|metricName → FQCN
  3. Compile filter expressions the same way
  4. Write manifest: META-INF/mal-v2-expression-map.properties

Rule file sources:

Generated class package: org.apache.skywalking.oap.meter.analyzer.v2.compiler.rt.*

LAL

Uses LALClassGenerator with setClassOutputDir() and setClassNameHint().

For each LAL YAML rule file (8 files, 10 rules):
  1. Parse YAML → List<LALConfig>
  2. For each rule:
     a. generator.setClassNameHint(sanitize(yamlSourceId + "_" + ruleName))
     b. generator.setYamlSource(yamlSourceId)
     c. generator.setExtraLogType(extraLogType)  // if needed
     d. LalExpression expr = generator.compile(dsl)
     e. .class file auto-written to classOutputDir

Generated class package: org.apache.skywalking.oap.log.analyzer.v2.compiler.rt.*

Hierarchy

Uses HierarchyRuleClassGenerator with setClassOutputDir() and setClassNameHint().

1. Load hierarchy-definition.yml
2. For each rule expression:
   a. generator.setClassNameHint(sanitize(ruleName))
   b. BiFunction result = generator.compile(ruleName, expression)
   c. .class file auto-written to classOutputDir

4 rule types: name, short-name, lower-short-name-remove-namespace, lower-short-name-with-fqdn.

Generated class package: org.apache.skywalking.oap.server.core.config.v2.compiler.hierarchy.rule.rt.*

Runtime Loading via Same-FQCN Replacements

At runtime (native image or JVM distro), same-FQCN replacement classes load pre-compiled classes instead of running Javassist.

OAL: OALEngineLoaderService

Upstream: server-core/.../oal/rt/OALEngineLoaderService.java Replacement: oap-libs-for-graalvm/server-core-for-graalvm/

On first load() call, reads manifests and registers all OAL classes:

• META-INF/oal-metrics-classes.txt → StreamAnnotationListener.notify()
• META-INF/oal-dispatcher-classes.txt → DispatcherDetectorListener.addIfAsSourceDispatcher()
• META-INF/oal-disabled-sources.txt → DisableRegister

MAL: DSL (v2)

Upstream: analyzer/meter-analyzer/.../v2/dsl/DSL.java Replacement: oap-libs-for-graalvm/meter-analyzer-for-graalvm/

parse(metricName, expression) loads per-file configs from META-INF/mal-v2/, looks up the pre-compiled class by expression text, and wires closure fields:

// At startup: load META-INF/mal-v2.manifest → per-file configs → expression→FQCN map
String className = expressionMap.get(expression);
MalExpression malExpr = (MalExpression) Class.forName(className)
    .getDeclaredConstructor().newInstance();
wireClosures(malExpr.getClass(), malExpr);  // LambdaMetafactory wiring
return new Expression(metricName, expression, malExpr);

Per-file manifests: META-INF/mal-v2/ mirrors the original YAML directory structure. Each config file (e.g., otel-rules/vm.yaml) contains rule names, full expressions, filter info, and compiled class FQCNs. META-INF/mal-v2.manifest lists all config files. Runtime builds an in-memory expression → FQCN map from all per-file configs. Since expressions are globally unique (different YAML files produce different expressions even for the same metric name), no hash or disambiguation is needed.

Closure wiring: Pre-compiled MalExpression classes may have public fields typed as functional interfaces (TagFunction, ForEachFunction, PropertiesExtractor, DecorateFunction). At build time, MALClassGenerator wires these via LambdaMetafactory after class loading. At runtime, the replacement DSL.java replicates this wiring by scanning public fields and invoking LambdaMetafactory.metafactory() to create the functional interface wrapper from the corresponding method (e.g., field _tag → method _tag_apply).

LAL: DSL (v2)

Upstream: analyzer/log-analyzer/.../v2/dsl/DSL.java Replacement: oap-libs-for-graalvm/log-analyzer-for-graalvm/

of(moduleManager, config, dsl, extraLogType, ruleName, yamlSource) computes the deterministic class name and loads via Class.forName().

Hierarchy: CompiledHierarchyRuleProvider

Upstream: analyzer/hierarchy/.../v2/compiler/CompiledHierarchyRuleProvider.java Replacement: oap-libs-for-graalvm/server-core-for-graalvm/

buildRules(ruleExpressions) loads pre-compiled BiFunction classes by rule name.

Classpath Scanning Replacements

Guava ClassPath.from() is used in several places. All replaced with manifest-based loading:

MeterSystem Javassist (Unchanged)

MeterSystem.create() generates one dynamic meter subclass per metric rule (~1188 classes) via Javassist. This is separate from MAL DSL compilation and is handled the same as before: run at build time, export .class files, load from manifests at runtime.

Manifest Files

All manifests are in META-INF/ within the precompiler output JAR:

Reflection Config Generation

The precompiler auto-generates reflect-config.json from manifests:

• OAL metrics/dispatchers: constructor-only access
• MAL/LAL/Hierarchy expression classes: constructor-only access
• Annotation-scanned classes: full method/field access
• Armeria HTTP handlers, GraphQL resolvers/types: full access

MAL v2 expressions use LambdaMetafactory for closure wiring. Since the pre-compiled classes are on the native-image build classpath, the static analysis resolves these lambdas automatically.

Same-FQCN Replacement Summary

Verification

# 1. Build precompiler and verify class generation
mvn -pl build-tools/precompiler install -DskipTests

# 2. Check generated v2 classes exist in output JAR
jar tf build-tools/precompiler/target/precompiler-*-generated.jar | grep "v2/compiler/rt"

# 3. Run tests
mvn -pl oap-graalvm-server test

# 4. Full distro build
make build-distro

# 5. Native image
make native-image