Let’s talk about extract, transform, and load, also known as ETL. If you are an AEM professional, this is something you have previously dealt with. It could be something along the lines of products, user bios, or store locations.

The extract and transform parts may differ depending on your source and requirements. The loading part is almost always going to be into AEM. While there may be a few ways to do that, let us talk about what is there for you out-of-the-box.

Sling Post Servlet

As an AEM developer, the Sling Post Servlet is something you should be familiar with. In particular, there is an import operation. This allows us to do the following:

curl -L https://www.boredapi.com/api/activity | \
curl -u admin:admin \
     -F":contentFile=@-" \
     -F":nameHint=activity" \
     -F":operation=import" \
     -F":contentType=json" \
     http://localhost:4502/content/mysite/us/en/jcr:content/root/container

You can run this many times. You will get activity_* nodes under /content/mysite/us/en/jcr:content/root/container. This assumes that the source is already in the format you desire. Meaning you have already done the transform part.

And the import operation can deal with more complex JSON structures, even XML. Here is a possible output that could be provided by a transform:

{
    "jcr:primaryType": "cq:Page",
    "jcr:content": {
        "jcr:primaryType": "cq:PageContent",
        "jcr:title": "My Page",
        "sling:resourceType": "mysite/components/page",
        "cq:template": "/conf/mysite/settings/wcm/templates/page-content",
        "root": {
            "jcr:primaryType": "nt:unstructured",
            "sling:resourceType": "mysite/components/container",
            "layout": "responsiveGrid",
            "container": {
                "jcr:primaryType": "nt:unstructured",
                "sling:resourceType": "mysite/components/container"
            }
        }
    }
}

Save this to a file named mypage.json and run the following curl command.

curl -u admin:admin \
     -F":name=my-page" \
     -F":contentFile=@mypage.json" \
     -F":operation=import" \
     -F":contentType=json" \
     -F":replace=true" \
     -F":replaceProperties=true" \
     http://localhost:4502/content/mysite/us/en

And boom! You have an instant page. This time instead of the :nameHint I used the :name and :replace properties. Running this command again will update the page. The loading part becomes really trivial and you need only worry about extracting and transforming.

Finding the Content Importer Service

While the Sling Post Servlet is well documented, its internal implementation is not. Luckily, it is open source. You won’t have to do any decompiling today! Let’s read the doPost function of the implementation. There are too many goodies we could dive into. Let’s stay focused. We are looking for the import operation. Did you find it?

You should have wound up at the doRun function of the ImportOperation.java. This is where all those request parameters from the curl commands above come into play. Go further down. You will find a call to ContentImporter.importContent(Node, String, String, InputStream, ImportOptions, ContentImportListener). Can you find its implementation?

Finally, you should have wound up on the DefaultContentImporter.java implementation. An OSGi component that implements the ContentImporter interface.

Programmatically Using the Content Importer

Yes! Programatically doing things. Now that we know that the ContentImporter is available as an OSGi component all we need is:

@Reference
private ContentImporter contentImporter;

And assuming you have your content via InputStream we can import the content under any node. As an example, I am using the SimpleServlet generated as part of the AEM Maven Archtype. I’m using Lombok to speed things up a little.

@Component(service = { Servlet.class })
@SlingServletResourceTypes(resourceTypes = "mysite/components/page", methods = HttpConstants.METHOD_GET, extensions = "txt")
@ServiceDescription("Simple Demo Servlet")
@Slf4j
public class SimpleServlet extends SlingSafeMethodsServlet {

    private static final long serialVersionUID = 1L;

    @Reference
    private ContentImporter contentImporter;

    @Override
    protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response)
            throws
            IOException {

        final MyContentImportListener contentImportListener = new MyContentImportListener();
        final Node node = request.getResource().adaptTo(Node.class);

        if (node != null) {

            final MyImportOptions importOptions = MyImportOptions.builder()
                                                                 .overwrite(true)
                                                                 .propertyOverwrite(true)
                                                                 .build();
            try (InputStream inputStream = IOUtils.toInputStream("{\"foo\":\"bar\"}", StandardCharsets.UTF_8)) {
                this.contentImporter.importContent(node, "my-imported-structure", "application/json", inputStream, importOptions, contentImportListener);
            } catch (final RepositoryException e) {
                log.error(e.getMessage(), e);
            }
        }

        response.setContentType("text/plain");
        response.getWriter().println(contentImportListener);

    }

    @Builder
    @Getter
    private static final class MyImportOptions extends ImportOptions {

        private final boolean checkin;
        private final boolean autoCheckout;
        private final boolean overwrite;
        private final boolean propertyOverwrite;

        @Override
        public boolean isIgnoredImportProvider(final String extension) { return false; }
    }

    @Getter
    @ToString
    private static final class MyContentImportListener implements ContentImportListener {

        private final com.google.common.collect.Multimap<String, String> changes =
                com.google.common.collect.ArrayListMultimap.create();

        @Override
        public void onReorder(final String orderedPath, final String beforeSibbling) {this.changes.put("onReorder", String.format("%s, %s", orderedPath, beforeSibbling)); }

        @Override
        public void onMove(final String srcPath, final String destPath) { this.changes.put("onMove", String.format("%s, %s", srcPath, destPath)); }

        @Override
        public void onModify(final String srcPath) { this.changes.put("onModify", srcPath); }

        @Override
        public void onDelete(final String srcPath) { this.changes.put("onDelete", srcPath); }

        @Override
        public void onCreate(final String srcPath) { this.changes.put("onCreate", srcPath); }

        @Override
        public void onCopy(final String srcPath, final String destPath) { this.changes.put("onCopy", String.format("%s, %s", srcPath, destPath)); }

        @Override
        public void onCheckin(final String srcPath) { this.changes.put("onCheckin", srcPath); }

        @Override
        public void onCheckout(final String srcPath) { this.changes.put("onCheckout", srcPath); }
    }
}

Conclusion

A Simple Example

Let us use this servlet as a real simple example.

@Component(service = { Servlet.class })
@SlingServletResourceTypes(resourceTypes = NameConstants.NT_PAGE,
                           methods = HttpConstants.METHOD_GET,
                           selectors = "proxy",
                           extensions = "json")
@Designate(ocd = ProxyServlet.Configuration.class)
public final class ProxyServlet extends SlingSafeMethodsServlet {

    private static final long serialVersionUID = -2678188253939985649L;

    @Activate
    private transient Configuration configuration;

    @Override
    protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response) throws IOException {

        response.setContentType("application/json");
        response.setCharacterEncoding("utf-8");

        final HttpUriRequest httpRequest = new HttpGet(this.configuration.uri());
        try (CloseableHttpClient httpClient = HttpClients.createDefault();
                CloseableHttpResponse httpResponse = httpClient.execute(httpRequest)) {

            httpResponse.getEntity()
                        .writeTo(response.getOutputStream());
        }
    }

    @ObjectClassDefinition
    public @interface Configuration {

        @AttributeDefinition String uri() default "https://randomuser.me/api/?results=1&inc=id,email,name,gender&seed=9579cb0f52986aab&noinfo";
    }
}

@ExtendWith(MockitoExtension.class)
class ProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    @ParameterizedTest
    @ValueSource(strings = { "https://someapi.com" })
    void doTest(final String uri) throws Exception {

        // mock osgi config
        doReturn(uri).when(this.configuration)
                     .uri();

        // mock http components
        final CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
        final HttpEntity httpEntity = mock(HttpEntity.class);
        doReturn(httpEntity).when(httpResponse)
                            .getEntity();

        final CloseableHttpClient httpClient = mock(CloseableHttpClient.class);
        final MockedConstruction.MockInitializer<HttpGet> mockInitializer = (mock, context) -> {
            assertEquals(uri, context.arguments().get(0));
            doReturn(httpResponse).when(httpClient)
                                  .execute(mock);
        };

        // mock http get and client construction
        try (MockedConstruction<HttpGet> construction = mockConstruction(HttpGet.class, mockInitializer);
                MockedStatic<HttpClients> mockedStatic = mockStatic(HttpClients.class)) {

            mockedStatic.when(HttpClients::createDefault)
                        .thenReturn(httpClient);

            // mock sling
            final SlingHttpServletRequest request = mock(SlingHttpServletRequest.class);
            final SlingHttpServletResponse response = mock(SlingHttpServletResponse.class);
            final ServletOutputStream outputStream = mock(ServletOutputStream.class);
            doReturn(outputStream).when(response)
                                  .getOutputStream();

            // execute
            this.servlet.doGet(request, response);

            // verify headers were set
            verify(response).setContentType("application/json");
            verify(response).setCharacterEncoding("utf-8");
            // verify uri was retreived from config
            verify(this.configuration).uri();
            // verify only one construction of http get
            assertEquals(1, construction.constructed()
                                        .size());
            // verify default client created
            mockedStatic.verify(HttpClients::createDefault);
            // verify get was executed by client
            verify(httpClient).execute(construction.constructed()
                                                   .get(0));
            // verify the entity was written to the output stream
            verify(httpEntity).writeTo(outputStream);
        }
    }
}

@Component(service = { Servlet.class })
@SlingServletResourceTypes(resourceTypes = NameConstants.NT_PAGE,
                           methods = HttpConstants.METHOD_GET,
                           selectors = "proxy",
                           extensions = "json")
@Designate(ocd = ProxyServlet.Configuration.class)
@Accessors(fluent = true,
           chain = true,
           prefix = "supply")
public final class ProxyServlet extends SlingSafeMethodsServlet {

    private static final long serialVersionUID = -2678188253939985649L;

    @Activate
    private transient Configuration configuration;

    @Setter(AccessLevel.PACKAGE)
    private transient Function<String, HttpUriRequest> supplyHttpRequest = HttpGet::new;
    @Setter(AccessLevel.PACKAGE)
    private transient Supplier<CloseableHttpClient> supplyHttpClient = HttpClients::createDefault;

    @Override
    protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response) throws IOException {

        response.setContentType("application/json");
        response.setCharacterEncoding("utf-8");

        final HttpUriRequest httpRequest = this.supplyHttpRequest.apply(this.configuration.uri());
        try (CloseableHttpClient httpClient = this.supplyHttpClient.get();
                CloseableHttpResponse httpResponse = httpClient.execute(httpRequest)) {

            httpResponse.getEntity()
                        .writeTo(response.getOutputStream());
        }
    }

    @ObjectClassDefinition
    public @interface Configuration {

        @AttributeDefinition String uri() default "https://randomuser.me/api/?results=1&inc=id,email,name,gender&seed=9579cb0f52986aab&noinfo";
    }
}

@ExtendWith(MockitoExtension.class)
class ProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    @ParameterizedTest
    @ValueSource(strings = { "https://someapi.com" })
    void doTest(final String uri) throws IOException {

        // mock osgi config
        doReturn(uri).when(this.configuration)
                     .uri();

        // mock http components
        final CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
        final HttpEntity httpEntity = mock(HttpEntity.class);
        doReturn(httpEntity).when(httpResponse)
                            .getEntity();

        final CloseableHttpClient httpClient = mock(CloseableHttpClient.class);
        final HttpUriRequest httpGet = mock(HttpUriRequest.class);
        doReturn(httpResponse).when(httpClient)
                              .execute(httpGet);

        // mock sling
        final SlingHttpServletRequest request = mock(SlingHttpServletRequest.class);
        final SlingHttpServletResponse response = mock(SlingHttpServletResponse.class);
        final ServletOutputStream outputStream = mock(ServletOutputStream.class);
        doReturn(outputStream).when(response)
                              .getOutputStream();

        // execute with suppliers
        this.servlet.httpClient(() -> httpClient)
                    .httpRequest(value -> {
                        assertEquals(uri, value);
                        return httpGet;
                    })
                    .doGet(request, response);

        // verify headers were set
        verify(response).setContentType("application/json");
        verify(response).setCharacterEncoding("utf-8");
        // verify uri was retreived from config
        verify(this.configuration).uri();
        // verify the entity was written to the output stream
        verify(httpEntity).writeTo(outputStream);
    }
}

@ExtendWith({ MockitoExtension.class, AemContextExtension.class })
class MockServerProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    private MockWebServer mockWebServer;

    @BeforeEach
    void beforeEach() throws IOException {

        this.mockWebServer = new MockWebServer();
        this.mockWebServer.start();
    }

    @AfterEach
    void afterEach() throws IOException {

        this.mockWebServer.shutdown();
    }

    @Test
    void doTest(final AemContext context) throws IOException {

        // setup
        final String url = this.mockWebServer.url("/").toString();
        doReturn(url).when(this.configuration).uri();
        this.mockWebServer.enqueue(new MockResponse().setBody("hello, world!"));

        // execute
        this.servlet.doGet(context.request(), context.response());

        // assert
        final String result = context.response().getOutputAsString();
        assertEquals("hello, world!", result);
    }
}

Note On Dependencies

In order to make the examples above work, I used an AEM Maven Archetype Project

mvn -B archetype:generate \
    -D archetypeGroupId=com.adobe.aem \
    -D archetypeArtifactId=aem-project-archetype \
    -D archetypeVersion=26 \
    -D appTitle="My Site" \
    -D appId="mysite" \
    -D groupId="com.mysite" \
    -D aemVersion=6.5.0

<!-- 1. update version of junit-bom -->
<dependency>
    <groupId>org.junit</groupId>
    <artifactId>junit-bom</artifactId>
    <!--<version>5.6.2</version>-->
    <version>5.7.0</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

<!-- 2. replace mockito-core with mockito-inline -->
<dependency>
    <groupId>org.mockito</groupId>
    <!--<artifactId>mockito-core</artifactId>
    <version>3.3.3</version>-->
    <artifactId>mockito-inline</artifactId>
    <version>3.11.2</version>
    <scope>test</scope>
</dependency>

<!-- 1. add the following two -->
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.20</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>com.squareup.okhttp3</groupId>
    <artifactId>mockwebserver</artifactId>
    <version>4.9.1</version>
    <scope>test</scope>
</dependency>

<!-- 2. remove junit-addons -->
<!-- <dependency>
    <groupId>junit-addons</groupId>
    <artifactId>junit-addons</artifactId>
    <scope>test</scope>
</dependency>-->

<!-- 3. replace mockito-core with mockito-inline -->
<dependency>
    <groupId>org.mockito</groupId>
    <!--<artifactId>mockito-core</artifactId>-->
    <artifactId>mockito-inline</artifactId>
    <scope>test</scope>
</dependency>

Tag ID Model

As an example, I am going to revisit the Tag id model from that first blog post. In that example, the multifield items had a tag ids property named ./tags. This I injected into the model as ItemTags

@ChildResource
ItemTags getTags();

interface ItemTags extends Iterable<Tag> {
}

@Model(adaptables = Resource.class, adapters = ItemTags.class)
final class ItemTagsImpl implements ItemTags {
    private final List<Tag> tags;
    @Inject
    public ItemTagsImpl(
                    @Self @Via("resourceResolver") final TagManager tagManager,
                    @Self final String[] ids) {

        this.tags = Arrays.stream(ids)
                          .map(tagManager::resolve)
                          .filter(Objects::nonNull)
                          .collect(Collectors.toUnmodifiableList());
    }
    @Override
    public Iterator<Tag> iterator() {
        return this.tags.iterator();
    }
    @Override
    public void forEach(final Consumer<? super Tag> action) {
        this.tags.forEach(action);
    }
    @Override
    public Spliterator<Tag> spliterator() {
        return this.tags.spliterator();
    }
}

@UtilityClass
public class TagUtil {

    public List<Tag> resolve(final TagManager tagManager, final String... ids) {

        return Arrays.stream(ids)
                     .map(tagManager::resolve)
                     .filter(Objects::nonNull)
                     .collect(Collectors.toUnmodifiableList());
    }
}

@ConsumerType
interface Tagged {
    List<Tag> getTags();
}
@ConsumerType
interface Item extends Tagged {
    String getTitle();
    Page getPage();
}

@Model(adapters = Tagged.class, adaptables = Resource.class)
class TaggedImpl implements Tagged {

    @Getter
    private List<Tag> tags;
    @ValueMapValue(name = "tags",
                   injectionStrategy = InjectionStrategy.OPTIONAL)
    private String[] ids;
    @Self
    @Via("resourceResolver")
    private TagManager tagManager;

    @PostConstruct
    public void activate() {

        this.tags = Arrays.stream(ArrayUtils.nullToEmpty(this.ids))
                          .map(this.tagManager::resolve)
                          .filter(Objects::nonNull)
                          .collect(Collectors.toUnmodifiableList());
    }
}

@Model(adapters = Item.class, adaptables = Resource.class)
final class ItemImpl extends TaggedImpl implements Item {

    @Getter
    @ValueMapValue(name = "jcr:title")
    private String title;
    @Getter
    @ResourcePath
    private Page page;
}

public interface TagService {

    List<Tag> resolve(TagManager tagManager, String... ids);
}

@Component(service = TagService.class)
public final class TagServiceImpl implements TagService {

    @Override
    public List<Tag> resolve(final TagManager tagManager, final String... ids) {

        return Arrays.stream(ids)
                     .map(tagManager::resolve)
                     .filter(Objects::nonNull)
                     .collect(Collectors.toUnmodifiableList());
    }
}

@Inject
public ItemTagsImpl(
        @Self @Via("resourceResolver")
        final TagManager tagManager,
        @OSGiService final TagService tagService,
        @Self final String[] ids) {

    this.tags = tagService.resolve(tagManager, ids);
}

// calling the injector by name
@Inject
@Source("valuemap")
private String name;

// or using the injector-specific annotation
@ValueMapValue
private String name;

@ServiceRanking(Integer.MAX_VALUE)
@Component(service = { Injector.class })
public final class TagIdInjector implements Injector {

    @Override
    public String getName() {

        return "tag-id";
    }

    @Override
    public Object getValue(final Object adaptable, final String name, final Type type, final AnnotatedElement element,
            final DisposalCallbackRegistry callbackRegistry) {

        final var tags = new ArrayList<Tag>();
        final var resource = (Resource) adaptable;
        final var ids = resource.getValueMap()
                                .get(name, String[].class);

        if (ids != null) {
            Optional.ofNullable(resource.getResourceResolver()
                                        .adaptTo(TagManager.class))
                    .ifPresent(tagManager -> Arrays.stream(ids)
                                                   .map(tagManager::resolve)
                                                   .filter(Objects::nonNull)
                                                   .forEach(tags::add));

        }

        return Collections.unmodifiableList(tags);
    }
}

@Model(adaptables = { Resource.class })
public interface MyModel {

    @ValueMapValue
    String getText();

    @ChildResource(injectionStrategy = InjectionStrategy.OPTIONAL)
    List<Item> getItems();

    @Model(adaptables = Resource.class)
    interface Item {

        @ValueMapValue(name = "jcr:title")
        String getTitle();

        @ResourcePath
        Page getPage();

        @Inject
        @Source("tag-id")
        List<Tag> getTags();
    }
}

AEM is a robust platform full of useful APIs and frameworks available at our disposal. Understanding what’s in the box will help us write less code. In my previous blog post, I covered one of the most used frameworks, Sling Models. Plus, I showed a real-world example with multi-fields. For this blog post, we will continue to leverage Sling Models with added help from a code generator.

Code Generators

Code generators save us from having to write monotonous Java plumbing code. Things like bean getters and setters, loggers, equals(), toString() and hashCode(). We need to free ourselves from having to write and maintain this plumbing. In doing so, we can focus on high-level business logic.

Some of the better-known generators are Immutables, AutoValue and Lombok. I’m not going to get into the details of each. Suffice it to say that after a few years and a few projects my personal preference is Lombok. Like the other two, it can generate value objects. But it does so much more than that.

javac, the Java compiler, executes code generators. Like all things in Java, there is a JSR. In this case JSR 269: Pluggable Annotation Processing API. Since we are using Maven, all we need to do is add the Maven dependency

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.20</version>
    <scope>provided</scope>
</dependency>

As if by magic, all your Lombok annotated classes will get augmented with generated code. Well… not magic. Javac will find and load META-INF/services/javax.annotation.processing.Processor. This file resides in lombok.jar and points to the appropriate annotation processor. You can read a more detailed explanation here. It is so seamless that the first time I pulled down a project built with Lombok I was not even aware of its presence. Not until I ran into an annotated class. That is on the compiler side. On the IDE side, the latest version of IntelliJ is already compatible with Lombok. Eclipse requires some setup.

A Real-World Example

At some point, every AEM developer has had to add classes to the <body> tag. Sounds simple right? We will add one constraint: properly extend the WCM Core Page Template Component. The kneejerk reaction would be to overwrite the page.html file where the <body> tag is. I did say “properly extend” right? That file has a lot of things in it to support Adobe’s platform & features. If we overwrite, then we would become responsible for keeping it up to date between upgrades.

The <body> classes are coming form the page model com.adobe.cq.wcm.core.components.models.Page. The proper way to extend the core model is through the ResourceSuperType, a Via provider type. Here is a more detailed example of how to extend core components.

Let’s fire up a new archetype project:

mvn -B archetype:generate \
    -D archetypeGroupId=com.adobe.aem \
    -D archetypeArtifactId=aem-project-archetype \
    -D archetypeVersion=26 \
    -D appTitle="My Site" \
    -D appId="mysite" \
    -D groupId="com.mysite" \
    -D aemVersion=6.5.5

And the first class we will create is com.mysite.core.models.MyPageModel

@Model(adaptables = { SlingHttpServletRequest.class }, adapters = { Page.class, ContainerExporter.class }, resourceType = "mysite/components/page")
@Exporter(name = ExporterConstants.SLING_MODEL_EXPORTER_NAME, extensions = ExporterConstants.SLING_MODEL_EXTENSION)
public final class MyPageModel implements Page, ContainerExporter {

    private static final Logger log = LoggerFactory.getLogger(MyPageModel.class);

    @Self
    @Via(type = ResourceSuperType.class)
    private Page delegate;

    @Override
    public String getCssClassNames() {
        log.info("this is the only relevant change");
        return String.join(" ", "foo", "bar", this.delegate.getCssClassNames());
    }

    /** Evertying below this is plumbing. */
    @Override
    public String getLanguage() { return this.delegate.getLanguage(); }

    @Override
    public Calendar getLastModifiedDate() { return this.delegate.getLastModifiedDate(); }

    @Override
    public String[] getKeywords() { return this.delegate.getKeywords(); }

    @Override
    public String getDesignPath() { return this.delegate.getDesignPath(); }

    @Override
    public String getStaticDesignPath() { return this.delegate.getStaticDesignPath(); }

    @Override
    public String getTitle() { return this.delegate.getTitle(); }

    @Override
    public String[] getClientLibCategories() { return this.delegate.getClientLibCategories(); }

    @Override
    public String[] getClientLibCategoriesJsBody() { return this.delegate.getClientLibCategoriesJsBody(); }

    @Override
    public String[] getClientLibCategoriesJsHead() { return this.delegate.getClientLibCategoriesJsHead(); }

    @Override
    public String getTemplateName() { return this.delegate.getTemplateName(); }

    @Override
    public String getAppResourcesPath() { return this.delegate.getAppResourcesPath(); }

    @Override
    public NavigationItem getRedirectTarget() { return this.delegate.getRedirectTarget(); }

    @Override
    public boolean hasCloudconfigSupport() { return this.delegate.hasCloudconfigSupport(); }

    @Override
    public Set<String> getComponentsResourceTypes() { return this.delegate.getComponentsResourceTypes(); }

    @Override
    public String[] getExportedItemsOrder() { return this.delegate.getExportedItemsOrder(); }

    @Override
    public Map<String, ? extends ComponentExporter> getExportedItems() { return this.delegate.getExportedItems(); }

    @Override
    public String getExportedType() { return this.delegate.getExportedType(); }

    @Override
    public String getMainContentSelector() { return this.delegate.getMainContentSelector(); }

    @Override
    public List<HtmlPageItem> getHtmlPageItems() { return this.delegate.getHtmlPageItems(); }

    @Override
    public String getId() { return this.delegate.getId(); }

    @Override
    public ComponentData getData() { return this.delegate.getData(); }
}

Getting Rid of the Boilerplate Code

/** static logger automatically generated. */
@Slf4j
@Model(adaptables = { SlingHttpServletRequest.class },
       adapters = { Page.class, ContainerExporter.class },
       resourceType = "mysite/components/page")
@Exporter(name = ExporterConstants.SLING_MODEL_EXPORTER_NAME,
          extensions = ExporterConstants.SLING_MODEL_EXTENSION)
public final class MyPageModel implements Page, ContainerExporter {

    /** getter automatically generated. */
    @Getter
    private String cssClassNames;

    /** delegate the core page model with exclusions. */
    @Delegate(excludes = MyDelegateExclusions.class)
    @Self
    @Via(type = ResourceSuperType.class)
    private Page delegate;

    @PostConstruct
    public void activate() {

        log.trace("component activation");
        this.cssClassNames = String.join(" ", "lombok", "foo", "bar", this.delegate.getCssClassNames());
    }

    /** signatures excluded from delegation. */
    private interface MyDelegateExclusions {

        /** we don't want this to get delegated. */
        String getCssClassNames();
    }
}

And the code (or lack thereof) pretty much speaks for itself. We got rid of the ubiquitous logger by adding @Slf4J at the class level. Yet our private static log is still available. We annotated cssClassNames with @Getter thereby implementing Page.getCssClassNames(). And of course, we got rid of all the delegated methods by annotating the delegate field with @Delegate.

What We Learned

For starters, I hope you learned there is a right way to extend core components and models. Second, we learned that Sling models and code generators are not mutually exclusive. Code generators can work in conjunction with Sling Models. You can also leverage them on your OSGi services and any Java class within your application. Take a look at Lombok’s stable features, and then the experimental ones.

Implementing static code analysis might seem like a daunting task. In some cases, this may be true depending on logistics, timing, and other factors. There is however a quick and easy way to implement it for AEM projects. Using open-source tools such as CheckStyle, SpotBugs, PMD, and JaCoCo you will pay nothing and reap all the benefits. By leveraging these tools you can:

• Ensure that everyone is following the same standards to keep your codebase uniform
• Catch common mistakes and known security vulnerabilities early
• Speed up code reviews as it acts as an automated reviewer before an actual reviewer gets down to it.
• Teach because years of using analyzers will turn coders into programmers.

Keep in mind that this is in the context of AEM development. Generally speaking, there are 3 languages. HTL, JavaScript, and Java. We will be focusing on the Java analysis here.

Let us start by creating an archetype project:

mvn -B archetype:generate \
 -D archetypeGroupId=com.adobe.aem \
 -D archetypeArtifactId=aem-project-archetype \
 -D archetypeVersion=25 \
 -D appTitle="My Site" \
 -D appId="mysite" \
 -D groupId="com.mysite" \
 -D aemVersion=6.5.5

Rule Files

Each scanner is going to need a custom set of rules. The easiest way to make them available to your build is by packaging them up into a jar file. In the archetype project you created, add a new folder called static-analysis. Inside of it create this POM file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mysite</groupId>
    <artifactId>static-analysis</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

</project>

Create a resources folder and inside the following 2 XML files:

• src/main/resources/my-checkstyle.xml
• src/main/resources/my-pmd.xml

For CheckStyle 8.41, you can download the default Sun coding conventions from here.

For PMD, you can use the following empty ruleset. We will get into configuring it later.

<ruleset name="Basic"
         xmlns="http://pmd.sourceforge.net/ruleset/2.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://pmd.sourceforge.net/ruleset/2.0.0 https://pmd.sourceforge.io/ruleset_2_0_0.xsd">
</ruleset>

Make sure to add static-analysis to the list of modules in the root POM. These files will now be available via a dependency to the static-analysis module. If you wish to make it available to other projects, you could publish this module to a Maven repository.

CheckStyle

CheckStyle scans your Java code and looks for style errors. Proper indentation, curly braces, etc. It helps to enforce coding standards. You can find the list of checks for CheckStyle 8.41 here.

On my AEM projects, I use the default Sun checks. The only changes that I make are:

• Setting the max property to 120 on the LineLength module and allowing the use of @SuppressWarnings in code

<module name="Checker">
  ...
  <module name="SuppressWarningsFilter"/>
  <module name="LineLength">
    <property name="fileExtensions" value="java"/>
    <property name="max" value="120"/>
  </module>
  ...
</module>

• Setting the scope property to protected on the JavadocVariable module

<module name="Checker">
  ...
  <module name="TreeWalker">
    ...
    <module name="JavadocVariable">
      <property name="scope" value="protected"/>
    </module>
    ...
  </module>
  ...
</module>

• Edit the core/pom.xml and add the following plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-checkstyle-plugin</artifactId>
    <version>3.1.2</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>checkstyle</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <skip>false</skip>
        <failsOnError>true</failsOnError>
        <consoleOutput>true</consoleOutput>
        <linkXRef>false</linkXRef>
        <configLocation>my-checkstyle.xml</configLocation>
        <includeResources>false</includeResources>
        <includeTestResources>false</includeTestResources>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>com.mysite</groupId>
            <artifactId>static-analysis</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>com.puppycrawl.tools</groupId>
            <artifactId>checkstyle</artifactId>
            <version>8.41</version>
        </dependency>
    </dependencies>
</plugin>

You can refer to the documentation for checkstyle:checktyle goal to see what each flag is doing. The important things to call out are:

• We pull in our static-analysis dependency.
• The configLocaion matches the config file in our static-analysis dependency.
• We pull in the desired version of CheckStyle via a dependency. It matches the Sun Checks file link from above for consistency.

Now, you can run mvn clean install on the root of the project and we get:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:checkstyle (default) on project mysite.core: An error has occurred in Checkstyle report generation.: Failed during checkstyle execution: There are 47 errors reported by Checkstyle 8.41 with my-checkstyle.xml ruleset. -> [Help 1]

CheckStyle is reporting 47 errors and the build has failed. You can attempt to fix them, update the rules file so they are excluded, or set the <failsOnError> plugin flag to false so we can move on to PMD.

To view the report, open the file at target/site/checkstyle.html.

PMD

Like CheckStyle, PMD will scan your Java code. Instead of checking on the code styling, it dives in to see what your program is doing. My favorite rule checks for cyclomatic complexity. On top of that, it also does CPD (copy-paste-detection). You can find the list of rules for PMD 6.32.0 here.

Earlier we created an empty my-pmd.xml ruleset file. Now, we will build it by referencing OOTB ruleset files that can be imported. Then, we selectively exclude those rules that you don’t want.

<ruleset name="Basic"
         xmlns="http://pmd.sourceforge.net/ruleset/2.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://pmd.sourceforge.net/ruleset/2.0.0 https://pmd.sourceforge.io/ruleset_2_0_0.xsd">

    <rule ref="category/java/bestpractices.xml">
        <!-- deprecated -->
        <exclude name="PositionLiteralsFirstInCaseInsensitiveComparisons"/>
        <exclude name="PositionLiteralsFirstInComparisons"/>
    </rule>
    <rule ref="category/java/codestyle.xml">
        <!-- deprecated -->
        <exclude name="AbstractNaming"/>
        <exclude name="AvoidFinalLocalVariable"/>
        <exclude name="AvoidPrefixingMethodParameters"/>
        <exclude name="ForLoopsMustUseBraces"/>
        <exclude name="IfElseStmtsMustUseBraces"/>
        <exclude name="IfStmtsMustUseBraces"/>
        <exclude name="MIsLeadingVariableName"/>
        <exclude name="SuspiciousConstantFieldName"/>
        <exclude name="VariableNamingConventions"/>
        <exclude name="WhileLoopsMustUseBraces"/>
        <!-- impractical -->
        <exclude name="AtLeastOneConstructor"/>
    </rule>
    <rule ref="category/java/design.xml">
        <!-- deprecated -->
        <exclude name="ModifiedCyclomaticComplexity"/>
        <exclude name="NcssConstructorCount"/>
        <exclude name="NcssMethodCount"/>
        <exclude name="NcssTypeCount"/>
        <exclude name="StdCyclomaticComplexity"/>
        <!-- impractical -->
        <exclude name="DataClass"/>
        <exclude name="LawOfDemeter"/>
        <exclude name="LoosePackageCoupling"/>
    </rule>
    <rule ref="category/java/documentation.xml"/>
    <rule ref="category/java/errorprone.xml">
        <!-- deprecated -->
        <exclude name="DataflowAnomalyAnalysis"/>
        <exclude name="DoNotCallSystemExit"/>
        <exclude name="InvalidSlf4jMessageFormat"/>
        <exclude name="LoggerIsNotStaticFinal"/>
        <!-- impractical -->
        <exclude name="BeanMembersShouldSerialize"/>
    </rule>
    <rule ref="category/java/multithreading.xml">
        <!-- deprecated -->
        <exclude name="UnsynchronizedStaticDateFormatter"/>
    </rule>
    <rule ref="category/java/performance.xml">
        <!-- deprecated -->
        <exclude name="AvoidUsingShortType"/>
        <exclude name="SimplifyStartsWith"/>
        <!-- impractical -->
        <exclude name="AvoidInstantiatingObjectsInLoops"/>
    </rule>
    <rule ref="category/java/security.xml"/>

</ruleset>

Most of the excluded rules are deprecated. We exclude them to silence the warnings. Otherwise, there are only really 6 that we exclude because they are impractical (or so I have found) in AEM projects. Otherwise, I run my projects with all the default rules OOTB.

Edit the core/pom.xml and add the following plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-pmd-plugin</artifactId>
    <version>3.13.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>pmd</goal>
                <goal>cpd</goal>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <skip>false</skip>
        <failOnViolation>true</failOnViolation>
        <printFailingErrors>true</printFailingErrors>
        <linkXRef>false</linkXRef>
        <rulesets>
            <ruleset>my-pmd.xml</ruleset>
        </rulesets>
        <targetJdk>11</targetJdk>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>com.mysite</groupId>
            <artifactId>static-analysis</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>net.sourceforge.pmd</groupId>
            <artifactId>pmd-core</artifactId>
            <version>6.32.0</version>
        </dependency>
        <dependency>
            <groupId>net.sourceforge.pmd</groupId>
            <artifactId>pmd-java</artifactId>
            <version>6.32.0</version>
        </dependency>
    </dependencies>
</plugin>

Now, you can run mvn clean install on the root of the project and we get:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-pmd-plugin:3.13.0:check (default) on project mysite.core: You have 32 PMD violations. For more details see: /mysite/core/target/pmd.xml -> [Help 1]

PMD is reporting 32 violations and the build has failed. You can attempt to fix them, update the ruleset file so they are excluded, or set the <failOnViolation> plugin flag to false so we can move on to SpotBugs.

To view the report, open the file at target/site/pmd.html.

SpotBugs

CheckStyle and PMD operate on source code files. SpotBugs on the other hand operates on the compiled byte code. As the name implies, it does a better job at detecting bugs than the other two. There are about 400 standard bug patterns.

Just like CheckStyle and PMD you can build custom filter files that can be used to exclude checks. In practice, I have found that the defaults work well enough and reasonably apply to AEM projects. We will just configure the Maven plugin.

Edit the core/pom.xml and add the following plugin:

<plugin>
    <groupId>com.github.spotbugs</groupId>
    <artifactId>spotbugs-maven-plugin</artifactId>
    <version>4.2.0</version>
    <configuration>
        <skip>false</skip>
        <failOnError>true</failOnError>
        <xmlOutput>true</xmlOutput>
        <xmlOutputDirectory>target/site</xmlOutputDirectory>
    </configuration>
    <executions>
        <execution>
            <id>spotbugs</id>
            <goals>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
    <dependencies>
        <dependency>
          <groupId>com.github.spotbugs</groupId>
          <artifactId>spotbugs</artifactId>
          <version>4.2.2</version>
        </dependency>
    </dependencies>
</plugin>

Now run mvn clean install on the root of the project and we get a build SUCCESS. No bugs spotted. Go take a look at the bug descriptions. Try to create a bug within a core bundle to see if SpotBugs picks it up.

To view the report, open the file at target/site/spotbugs.xml.

JaCoCo

JaCoCo is the reporting tool that measures code coverage and generates visual reports. It relies on unit tests to execute your lines of code, collecting metrics along the way.

Edit the core/pom.xml and add the following plugin

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.6</version>
    <executions>
        <execution>
            <id>default-prepare-agent</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>default-report</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
        <execution>
            <id>default-check</id>
            <goals>
                <goal>check</goal>
            </goals>
            <configuration>
                <rules>
                    <rule>
                        <element>CLASS</element>
                        <limits>
                            <limit>
                                <counter>LINE</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>0.90</minimum>
                            </limit>
                            <limit>
                                <counter>BRANCH</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>0.90</minimum>
                            </limit>
                        </limits>
                    </rule>
                </rules>
            </configuration>
        </execution>
    </executions>
</plugin>

In the configuration, I have set the LINE and BRANCH coverage to at least 90%. Check out the other available counters here and the rule configuration here.

Now, you can run mvn clean install on the root of the project and we get a build SUCCESS. The archetype’s coverage is pretty high. To cause the rule to fail delete one of the unit tests.

To view the report, open the file at target/site/jacoco/index.html.

Conclusion

I love writing code. What I love more is writing less. Less code means fewer unit tests because you have much less coverage to hit. Well organized, modularized, short, and concise. How do you achieve this? Good software engineering practices, code generators, and leveraging existing frameworks and APIs.

Leveraging Frameworks and APIs

As an AEM developer, you are working within a platform that comes with tons of OOTB frameworks and APIs. Visit the list of OSGi bundles at http://localhost:4502/system/console/bundles. For example, on a recent visit (AEM 6.5.5.0), I noticed Commons BeanUtils 1.8.2. I’ve yet to find a use case for it, but I made a mental note and filed it for a later day.

Sling Models

One of the better-known frameworks is the Sling Models framework. “Better-known” in that we all know the simple use cases. However, few know what is going on under the hood. Enough to merit its own blog post that’s for sure. In the context of this one, this is one of the frameworks to help achieve our goal.

Let’s take the simple use case of the AEM Granite Multifield. It seems every project I’ve worked on has tons of code to read and write multifield data. There are 3 major reasons:

<items jcr:primaryType="nt:unstructured">
    <text
        jcr:primaryType="nt:unstructured"
        sling:resourceType="granite/ui/components/coral/foundation/form/textfield"
        fieldLabel="Text"
        name="./text"/>
    <multifield
        jcr:primaryType="nt:unstructured"
        sling:resourceType="granite/ui/components/coral/foundation/form/multifield"
        composite="{Boolean}true">
        <field
            jcr:primaryType="nt:unstructured"
            sling:resourceType="granite/ui/components/coral/foundation/container"
            name="./items">
            <items jcr:primaryType="nt:unstructured">
                <title
                    jcr:primaryType="nt:unstructured"
                    sling:resourceType="granite/ui/components/coral/foundation/form/textfield"
                    fieldLabel="Title"
                    name="./jcr:title"/>
                <page
                    jcr:primaryType="nt:unstructured"
                    sling:resourceType="granite/ui/components/coral/foundation/form/pathfield"
                    fieldLabel="Page"
                    name="./page"/>
                <tags
                    jcr:primaryType="nt:unstructured"
                    sling:resourceType="cq/gui/components/coral/common/form/tagfield"
                    fieldLabel="Tags"
                    multiple="{Boolean}true"
                    name="./tags"/>
            </items>
        </field>
    </multifield>
</items>

@Model(adaptables = { Resource.class })
public interface MyModel {

    @ValueMapValue
    String getText();

    @ChildResource(injectionStrategy = InjectionStrategy.OPTIONAL)
    List<Item> getItems();

    @Model(adaptables = Resource.class)
    interface Item {

        @ValueMapValue(name = "jcr:title")
        String getTitle();

        @ResourcePath
        Page getPage();

        @ValueMapValue
        List<String> getTags();
    }
}

Resource tags = resource.getChild("items/item0/tags");
String[] value = tags.adaptTo(String[].class);

@Model(adaptables = { Resource.class })
public interface MyModel {

    @ValueMapValue
    String getText();

    @ChildResource(injectionStrategy = InjectionStrategy.OPTIONAL)
    List<Item> getItems();

    @Model(adaptables = Resource.class)
    interface Item {

        @ValueMapValue(name = "jcr:title")
        String getTitle();

        @ResourcePath
        Page getPage();

        @ChildResource
        ItemTags getTags();
    }

    interface ItemTags extends Iterable<Tag> {

    }

    @Model(adaptables = Resource.class,
           adapters = ItemTags.class)
    final class ItemTagsImpl implements ItemTags {

        private final List<Tag> tags;

        @Inject
        public ItemTagsImpl(
                @Self
                @Via("resourceResolver")
                final TagManager tagManager,
                @Self
                final String[] ids) {

            this.tags = Arrays.stream(ids)
                              .map(tagManager::resolve)
                              .filter(Objects::nonNull)
                              .collect(Collectors.toUnmodifiableList());
        }

        @Override
        public Iterator<Tag> iterator() {

            return this.tags.iterator();
        }

        @Override
        public void forEach(final Consumer<? super Tag> action) {

            this.tags.forEach(action);
        }

        @Override
        public Spliterator<Tag> spliterator() {

            return this.tags.spliterator();
        }
    }
}

What is CORS

CORS stands for cross-origin resource sharing. It is a mechanism by which the server will control access to its goodies, should that someone be running on a different domain.

• It occurs between the browser and a server (usually some sort of API endpoint).
• The browser sends some information via HTTP Access-Control-Request-* headers.
• The server makes a determination on headers received a sends back Access-Control-Allow-* headers.
• The browser now knows if it can or can not access resources on this server.
• In some circumstances, the browser may do a pre-flight which is a test prior to doing the real request.
• The browser will not allow front-end code to modify the headers.
• The server-side code can change the headers. But, it should be transparent to the application and done by a downstream service such as an API gateway or the HTTP server.

The Access-Control-Request-* CORS headers tell the server

• My current domain (Origin header)
• HTTP method of the request (Access-Control-Request-Method header)
• List of headers in the request (Access-Control-Request-Headers header)

The Access-Control-Allow-* CORS headers tell the browser

• The origin that is allowed (Access-Control-Allow-Origin header)
• The methods that are allowed (Access-Control-Allow-Methods header)

This is the heart of CORS. If the backend service does not send back Access-Control-Allow-* headers with correct values, the browser will not allow the request to continue.

I like to think of the entire exchange as a Gentlemen’s Agreement. The gentlemen are the browser and the server. Why? Because it is an informal exchange of data that depends on the honesty of both parties. You can break the agreement simply by fudging with the headers.

Bypassing CORS

All we need to do is fool the browser and/or the service so that the AJAX request can proceed. Perhaps your browser has security switches you can flip. You can probably find a plugin to do the trick. Adding a host file entry so you can run your local site on the allowed domain may work. The best way I could think of would be to set up a proxy server to sit between the front-end code and back-end services.

Always be wary of proxy servers. After all, they can inspect your traffic and make changes to your payloads. In our case, we are going to toggle the required CORS headers to make the browser and API happy. And we will do this from within the safety of a local Docker container so we will never actually send out any data.

Setting Up A Service

If you already have a backend service with CORS enabled, great! For this test, we are going to set up a quick and dirty express service.

You will need to add this to your hosts’ file:

• 127.0.0.1 mybackend.com

The server.js file

const express = require('express');
var path = require('path');
const cors = require('cors');
const corsOpts = {
  origin: /myfrontend.com:3000/,
  allowedHeaders: []
}
const app = express();
app.get('/service', cors(corsOpts), (req, res) => {
        res.send(`hello, ${req.get('origin')}`);
        console.log(JSON.stringify(req.headers, null, '  '));
    })
   .get('/', cors(corsOpts), (req, res) => {
        res.sendFile(path.join(__dirname + '/index.html'));
    })
   .options('*', cors(corsOpts));

const server = app.listen(3000, () => {
    console.log(`server listening on port ${server.address().port}`);
});

The index.html file

<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <title>CORS</title>
</head>

<body>
    Response: <span id="response"></span>
</body>

<script type="text/javascript" src="//code.jquery.com/jquery-3.5.1.min.js"></script>
<script>
    $(function() {
        $.ajax({
            url: 'http://mybackend.com:3000/service',
            success: res => {
                $('#response').text(res);
            },
            error: err => {
                $('#response').text(err.statusText);
            }
        })
    });
</script>

</html>

And when you fire it up, you can access http://localhost:3000/. Notice the annoying CORS error in the browser’s console

Access to XMLHttpRequest at 'http://mybackend.com:3000/service' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Luckily, CORS errors on the browser are self-explanatory. The browser is saying, “I can’t allow this because the server did not inform me that the current origin is allowed to make this request.”

I mentioned earlier there are a few fixes:

• You can add 127.0.0.1 myfrontend.com to your hosts’ file and access the site via http://myfrontend.com:3000/ but editing the hosts’ file can be impractical or sometimes impossible.
• You can change server.js to allow localhost however we do not always have control over the services we integrate with.
• You can install a browser plugin such as Allow CORS: Access-Control-Allow-Origin in Chrome.
• You can add CORS Anywhere into your project if you have JavaScript skills.

Setting up an NGINX Proxy

For my particular project, Docker was already part of the local developer environment. All I needed to do was add an extra service to the docker-compose file and a couple of config files. The proxy sat there, transparent to developers. On the local run mode, the font-end code would integrate with the remote service via the local proxy. On all other run modes, it would integrate with the remote service.

The docker-compose.yml file

version: '3.8'

services:

  proxy:
    image: nginx:alpine
    ports:
      - 8080:80
    volumes:
      - ./default.conf.template:/etc/nginx/templates/default.conf.template

The default.conf.template file:

server {
    listen 80 default_server;
    server_name _;
    server_name_in_redirect off;

    access_log  /var/log/nginx/access.log;
    error_log  /var/log/nginx/error.log debug;

    location /service {
        proxy_pass http://host.docker.internal:3000;
    }
}

Make sure to update the URL on the JavaScript so it uses the proxy running on 8080:

$(function() {
    $.ajax({
        //url: 'http://mybackend.com:3000/service',
        url: 'http://mybackend.com:8080/service',
        success: res => {
            $('#response').text(res);
         },
        error: err => {
            $('#response').text(err.statusText);
        }
    })
});

Fire up the docker-compose service and access http://localhost:8080/. You still see the same CORS error but now to the proxied service on port 8080.

Modifying Requests and Responses

The error message is saying you are missing the Access-Control-Allow-Origin header. The service will return this header if the Origin matches the pattern /myfrontend.com:3000/. Let’s fool the service by supplanting the Origin header before sending it upstream

location /service {
    proxy_pass http://host.docker.internal:3000;
    proxy_set_header Origin http://myfrontend.com:3000;
}

The service will send back the Access-Control-Allow-Origin header and the CORS error changes

Access to XMLHttpRequest at 'http://mybackend.com:8080/service' from origin 'http://localhost:3000' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'http://myfrontend.com:3000' that is not equal to the supplied origin.

The browser is saying the service will only allow http://myfrontend.com:3000 and you are on localhost. Let’s fool the browser by supplanting the Access-Control-Allow-Origin header.

location /service {
    proxy_pass http://host.docker.internal:3000;
    proxy_set_header Origin http://myfrontend.com:3000;
    proxy_hide_header Access-Control-Allow-Origin;
    add_header Access-Control-Allow-Origin $http_origin;
}

First, we hide it. Then we add our own before sending back the response. NGINX will not replace headers; it will append to them. If you don’t hide it first, you’ll wind up with 2 origins and this will cause another CORS error. Like I said before, CORS errors on the browser are self-explanatory. Try it.

Something to Try on Your Own

I mentioned earlier pre-flight requests. To trigger one, update the AJAX call to add a header.

$(function() {
    $.ajax({
        headers: { 'X-Foo': 'bar' },
        url: 'http://mybackend.com:8080/service',
        success: res => {
            $('#response').text(res);
         },
        error: err => {
            $('#response').text(err.statusText);
        }
    })
});

In this example, we will add the X-Foo header. You’ll note that our service implementation allows no extra headers. The correct thing would be to update the service implementation. If you have no control over that you can leverage the proxy.

Look at the CORS error on the browser. Can you update the response at the proxy to fool the browser?

Conclusion

We set up a simple example to fool the browser and service into allowing the cross-origin request. Depending on your particular needs you may need to make extra changes. At the very least you now have a testing app and proxy.

Recently, I had a task to shorten URLs and remove the HTML extension in AEM. On top of that, I had to add a context root to the URLs so that these mappings would happen.

• /content/myapp/en_US.html -> /mycontext/
• /content/myapp/en_US/homepage.html -> /mycontext/homepage/
• /content/myapp/es_ES.html -> /mycontext/es_ES/
• /content/myapp/es_ES/homepage.html -> /mycontext/es_ES/homepage/

Shorting URLs in AEM consists of two things:

• The dispatcher will receive requests for short extension-less URLs and must rewrite them before passing them through to the publishers.
• The links within the HTML markup rendered by AEM must be rewritten to their short extension-less versions.

Setting up Apache is straight forward. You need to inspect the incoming request and translate it to its long version before sending it on to AEM. What I had trouble with was configuring the Sling mappings to rewrite URLs. There are 2 ways to achieve this:

• Update the content under /etc/map
• Add the mapping to the resource.resolver.mapping property of the org.apache.sling.jcr.resource.internal.JcrResourceResolverFactoryImpl OSGi configuration

What I Do

In general, I have always opted for the second. I find it is simpler to manage than managing content under /etc/map. The only annoying thing is that if you make a change it causes every OSGi component that depends on it to restart. Having to wait a long time between changes is frustrating. This is especially true if I am trying to work out new mapping rules.

Nowadays, I find myself doing more and more TDD. In particular, I’ve been leveraging wcm.io Sling Mocks. With JUnit 5 and AssertJ, I can develop Sling Models and OSGi components without ever having to load a page.

And this is it! I wrote the unit test and then I focused on creating a set of outgoing mappings. Once satisfied, I popped them into the OSGi configuration. Then, I sat around for 7 minutes until the system became responsive again, and the job was complete!

import static org.assertj.core.api.Assertions.assertThat;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.sling.api.resource.ResourceResolver;
import org.apache.sling.testing.mock.sling.ResourceResolverType;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import com.google.common.collect.ImmutableMap;
import io.wcm.testing.mock.aem.junit5.*;

@ExtendWith(AemContextExtension.class)
class UrlMappingTest {

    private static final ImmutableMap<String, Object> PROPERTIES =
        ImmutableMap.of("resource.resolver.mapping", ArrayUtils.toArray(
            "/:/",
            "^/content/myapp/en_US\\.html</mycontext/",
            "^/content/myapp/en_US/(.+)\\.html</mycontext/$1/",
            "^/content/myapp/(.+)\\.html</mycontext/$1/"
        ));

    private final AemContext context =
            new AemContextBuilder().resourceResolverType(ResourceResolverType.JCR_MOCK)
                                   .resourceResolverFactoryActivatorProps(PROPERTIES)
                                   .build();

    @Test
    public void testShortExtensionlessMappings() {

        final ResourceResolver resourceResolver = this.context.resourceResolver();

        final String defaultLocaleRoot =
            resourceResolver.map("/content/myapp/en_US.html");
        assertThat(defaultLocaleRoot).isEqualTo("/mycontext/");

        final String defaultLocalePath =
            resourceResolver.map("/content/myapp/en_US/hello-world.html");
        assertThat(defaultLocalePath).isEqualTo("/mycontext/hello-world/");

        final String localeRoot =
            resourceResolver.map("/content/myapp/es_US.html");
        assertThat(localeRoot).isEqualTo("/mycontext/es_US/");

        final String localePath =
            resourceResolver.map("/content/myapp/es_US/hello-world.html");
        assertThat(localePath).isEqualTo("/mycontext/es_US/hello-world/");

        final String qsParams =
            resourceResolver.map("/content/myapp/es_US/hello-world.html?foo=bar");
        assertThat(qsParams).isEqualTo("/mycontext/es_US/hello-world/?foo=bar");
    }
}

I hope you found this blog post useful. For any additional questions, please comment below and be sure to check out our other Adobe blogs.

Reverse proxy and caching are two of the many features provided by the AEM dispatcher module. It is the prism through which your web users will view content on the AEM publisher. Developers will say the development and operations (DevOps) team handles it. DevOps rely on developers to provide instructions on how to configure it. Hardly anyone sets it up in their local environment. Instead, they depend on remote systems to test and debug. This can sometimes turn into a game of whack-a-mole until things work. 

In this post, I will review how to set up a local dispatcher using docker to proxy to your local AEM publish instance. We will be able to update config files, trace logs, and debug. When we send the configuration to DevOps, it will be with an “it worked on my computer” stamp of approval. 

Install Docker 

We leverage Docker to avoid the difficulties that come with installing Apache on your platform (Windows or macOS). We are also getting an environment that is pretty close to what AMS would be using. Installing Docker on your platform should be fairly straight forward. Once installed, you can verify with the following commands

Create an Archetype Project 

You may have your own Apache configuration. Before you throw those in, let’s build a baseline using the archetype project. Note that the aemVersion is set to 6.5.0 which is NOT the same if you were using AEM as a cloud service. This means that they will work on AMS dispatchers, not AEM as a Cloud Service.

mvn -B archetype:generate \
 -D archetypeGroupId=com.adobe.aem \
 -D archetypeArtifactId=aem-project-archetype \
 -D archetypeVersion=24 \
 -D appTitle="My Site" \
 -D appId="mysite" \
 -D groupId="com.mysite" \
 -D aemVersion="6.5.0"

Building an Image 

To build an image, we are going to leverage an existing CentOS image. We will install Apache and the dispatcher module. This is pretty much the same setup on AMS. The only difference is that they use the Enterprise RedHat Linux which CentOS is the free version of. First, create the following Dockerfile within the mysite project folder: 

FROM centos:7.8.2003
ARG MODULE_URL

RUN yum -y update && \
    yum -y install httpd && \
    yum clean all && \
    mv $(curl --insecure --silent ${MODULE_URL} | \
    tar xzvf - --wildcards "*.so" | head -1) /etc/httpd/modules/mod_dispatcher.so && \
    chown root:root /etc/httpd/modules/mod_dispatcher.so

EXPOSE 80

RUN mkdir -p /mnt/var/www/author && \
    mkdir -p /mnt/var/www/html && \
    mkdir -p /mnt/var/www/default && \
    chown -R apache:apache /mnt/var/www

WORKDIR /etc/httpd
CMD [ "apachectl", "-DFOREGROUND" ]

Now, we build the image with the following command within the mysite project folder: 

docker build \
    -t mysite/dispatcher:latest \
    --build-arg MODULE_URL=https://download.macromedia.com/dispatcher/download/dispatcher-apache2.4-linux-x86_64-4.3.3.tar.gz \
    .

Creating a Container 

Next, you will need to create a container using the image by leveraging Docker Compose. Create a file called docker-compose.yml within the mysite folder. 

version: "3.8"

services:

  dispatcher:
    image: mysite/dispatcher:latest
    ports:
        - 80:80
    environment:
      DISP_ID: dispatcher1docker
      AUTHOR_DEFAULT_HOSTNAME: author.mycompany.local
      AUTHOR_IP: host.docker.internal
      AUTHOR_PORT: 4502
      AUTHOR_DOCROOT: /mnt/var/www/author
      PUBLISH_DEFAULT_HOSTNAME: mysite.mycompany.local
      PUBLISH_IP: host.docker.internal
      PUBLISH_PORT: 4503
      PUBLISH_DOCROOT: /mnt/var/www/html
      CRX_FILTER: deny
    volumes:
      - ./dispatcher/src/conf:/etc/httpd/conf:ro
      - ./dispatcher/src/conf.d:/etc/httpd/conf.d:ro
      - ./dispatcher/src/conf.dispatcher.d:/etc/httpd/conf.dispatcher.d:ro
      - ./dispatcher/src/conf.modules.d:/etc/httpd/conf.modules.d:ro
      - ./httpd-docroot:/mnt/var/www/html

What we are setting up here is a service called dispatcher. The environment variables represent the minimal set used by the archetype Apache configs. The volumes section maps each config folder to its place within the container. The last volume maps the HTML folder where the dispatcher is caching pages to our host’s directory. This is how they will persist should the container restart. It also allows us to navigate the cached data within our host’s file system. 

The archetype configs will create 2 vhosts. They are author.mycompany.local and mysite.mycompany.local. To access these locally, you should make sure to update your hosts file (i.e. /etc/hosts in macOS) so that they resolve to 127.0.0.1. 

Testing MySite Through the Default Dispatcher 

Start your publisher and deploy mysite by using the command we all know and love: 

mvn clean install -P autoInstallPackagePublish -P adobe-public

Start the dispatcher 

docker-compose up -d

And verify the following things 

• You can access mysite directly on the publisher: http://localhost:4503/content/mysite/us/en.html 
• You can access mysite through the dispatcher: http://mysite.mycompany.local/content/mysite/us/en.html 
• You are seeing things getting cached in the mapped folder, in our case httpd-docroot 

Things to Try Out 

There are many dispatcher related topics we could touch upon that would not fit into this tutorial. Here are a few ideas you can try out now that you have a working dispatcher. 

• Test the dispatcher flush 
• Remove the /content/mysite root by updating the rewrite rules 
• Create a new vhost for a new site 
• Access the author through the dispatcher 
• Set up Sling Server Side Includes 
• Tail the logs generated by Apache 
• Explore the modules under /etc/httpd/modules 

I hope you found this blog post useful. For any additional questions, please comment below and be sure to check out our other Adobe blogs.