Adobe

Mocking HTTP Clients in AEM Projects

Computercodebetter

The first thing I learned as an AEM developer was mvn clean install -P autoInstallBundle. The second thing was how to debug a remote Java process. The first allowed me to deploy updates I made to the bundle. The second allowed me to step through and debug the code. This was the normal cadence.

Since then, the AEM Maven Archetype has evolved to include JUnit 5 with wcm.io AEM Mocks. That sits on top of the Sling Mock APIs. With JUnit and the AEM Mocks, I have been able to switch from a build/deploy/debug cadence to a test-driven one. This includes resources, servlets, pages, tags, context-aware configurations, models, and OSGi components. You name it. You can hit about 90% of the use cases you might encounter during AEM OSGi bundle development.

This blog is not about 90%. I’m sure you can find a lot of examples out there. This is about that 10%, in particular with HTTP clients. Right out of the box, AEM comes with the Apache HTTP Client. If you are running AEM under Java 11, its HttpClient is an option as well. Either way, you face the same problem. How do you mock constructors and static classes?

A Simple Example

Let us use this servlet as a real simple example.

@Component(service = { Servlet.class })
@SlingServletResourceTypes(resourceTypes = NameConstants.NT_PAGE,
                           methods = HttpConstants.METHOD_GET,
                           selectors = "proxy",
                           extensions = "json")
@Designate(ocd = ProxyServlet.Configuration.class)
public final class ProxyServlet extends SlingSafeMethodsServlet {

    private static final long serialVersionUID = -2678188253939985649L;

    @Activate
    private transient Configuration configuration;

    @Override
    protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response) throws IOException {

        response.setContentType("application/json");
        response.setCharacterEncoding("utf-8");

        final HttpUriRequest httpRequest = new HttpGet(this.configuration.uri());
        try (CloseableHttpClient httpClient = HttpClients.createDefault();
                CloseableHttpResponse httpResponse = httpClient.execute(httpRequest)) {

            httpResponse.getEntity()
                        .writeTo(response.getOutputStream());
        }
    }

    @ObjectClassDefinition
    public @interface Configuration {

        @AttributeDefinition String uri() default "https://randomuser.me/api/?results=1&inc=id,email,name,gender&seed=9579cb0f52986aab&noinfo";
    }
}
It is your standard run-of-the-mill OSGi R7 servlet. It fetches data from some other API and proxies it back. If you have a keen eye, you have already spotted the problems. The HttpGet constructor and the HttpClients utility class. Half of you out there are thinking PowerMock. The other half are thinking about refactoring to make the code better suited for unit tests.

Static Mocking

You might have heard: static mocking is an anti-pattern. It is, if you are on the Spring framework. That relies on dependency injection. AEM instead relies on the Felix OSGi container. At most, OSGi components can inject other components. And Sling Model injection has limitations. I prefer to stay away from static mocking. I will refactor to a certain extent until I reach the point of diminishing returns. That is when the amount of code I have to write to support testing becomes too abstract or hard to maintain. Also, if you were considering PowerMock, consider Mockito. As of 3.4.0 there is static method mocking support and as of 3.5.0 there is constructor mocking support. Below is how I mocked up our simple servlet. Unlike PowerMock, Mockito has a JUnit 5 extension MockitoExtension.

@ExtendWith(MockitoExtension.class)
class ProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    @ParameterizedTest
    @ValueSource(strings = { "https://someapi.com" })
    void doTest(final String uri) throws Exception {

        // mock osgi config
        doReturn(uri).when(this.configuration)
                     .uri();

        // mock http components
        final CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
        final HttpEntity httpEntity = mock(HttpEntity.class);
        doReturn(httpEntity).when(httpResponse)
                            .getEntity();

        final CloseableHttpClient httpClient = mock(CloseableHttpClient.class);
        final MockedConstruction.MockInitializer<HttpGet> mockInitializer = (mock, context) -> {
            assertEquals(uri, context.arguments().get(0));
            doReturn(httpResponse).when(httpClient)
                                  .execute(mock);
        };

        // mock http get and client construction
        try (MockedConstruction<HttpGet> construction = mockConstruction(HttpGet.class, mockInitializer);
                MockedStatic<HttpClients> mockedStatic = mockStatic(HttpClients.class)) {

            mockedStatic.when(HttpClients::createDefault)
                        .thenReturn(httpClient);

            // mock sling
            final SlingHttpServletRequest request = mock(SlingHttpServletRequest.class);
            final SlingHttpServletResponse response = mock(SlingHttpServletResponse.class);
            final ServletOutputStream outputStream = mock(ServletOutputStream.class);
            doReturn(outputStream).when(response)
                                  .getOutputStream();

            // execute
            this.servlet.doGet(request, response);

            // verify headers were set
            verify(response).setContentType("application/json");
            verify(response).setCharacterEncoding("utf-8");
            // verify uri was retreived from config
            verify(this.configuration).uri();
            // verify only one construction of http get
            assertEquals(1, construction.constructed()
                                        .size());
            // verify default client created
            mockedStatic.verify(HttpClients::createDefault);
            // verify get was executed by client
            verify(httpClient).execute(construction.constructed()
                                                   .get(0));
            // verify the entity was written to the output stream
            verify(httpEntity).writeTo(outputStream);
        }
    }
}
My silly little servlet is like 7 lines of relevant code and my unit test is 3 times that! I didn’t even get into authentication, headers, query string parameters, or serialization.

Writing Testable Code

Even though the static mocking worked, the unit test was big and it is bound to get harder to maintain. Let’s consider some refactoring to make our servlet somewhat more testable.

@Component(service = { Servlet.class })
@SlingServletResourceTypes(resourceTypes = NameConstants.NT_PAGE,
                           methods = HttpConstants.METHOD_GET,
                           selectors = "proxy",
                           extensions = "json")
@Designate(ocd = ProxyServlet.Configuration.class)
@Accessors(fluent = true,
           chain = true,
           prefix = "supply")
public final class ProxyServlet extends SlingSafeMethodsServlet {

    private static final long serialVersionUID = -2678188253939985649L;

    @Activate
    private transient Configuration configuration;

    @Setter(AccessLevel.PACKAGE)
    private transient Function<String, HttpUriRequest> supplyHttpRequest = HttpGet::new;
    @Setter(AccessLevel.PACKAGE)
    private transient Supplier<CloseableHttpClient> supplyHttpClient = HttpClients::createDefault;

    @Override
    protected void doGet(final SlingHttpServletRequest request, final SlingHttpServletResponse response) throws IOException {

        response.setContentType("application/json");
        response.setCharacterEncoding("utf-8");

        final HttpUriRequest httpRequest = this.supplyHttpRequest.apply(this.configuration.uri());
        try (CloseableHttpClient httpClient = this.supplyHttpClient.get();
                CloseableHttpResponse httpResponse = httpClient.execute(httpRequest)) {

            httpResponse.getEntity()
                        .writeTo(response.getOutputStream());
        }
    }

    @ObjectClassDefinition
    public @interface Configuration {

        @AttributeDefinition String uri() default "https://randomuser.me/api/?results=1&inc=id,email,name,gender&seed=9579cb0f52986aab&noinfo";
    }
}

Here, I am leveraging Lombok. If you have read my previous post Writing Less Java Code in AEM with Sling Models & Lombok, you know I love it. It did shave off a few lines of boilerplate code. In this particular case, this is as far as I would go with refactoring. The functional interfaces that provide the HttpClient and HttpGet are good enough. Anything else would get too convoluted. Below is the unit test.

@ExtendWith(MockitoExtension.class)
class ProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    @ParameterizedTest
    @ValueSource(strings = { "https://someapi.com" })
    void doTest(final String uri) throws IOException {

        // mock osgi config
        doReturn(uri).when(this.configuration)
                     .uri();

        // mock http components
        final CloseableHttpResponse httpResponse = mock(CloseableHttpResponse.class);
        final HttpEntity httpEntity = mock(HttpEntity.class);
        doReturn(httpEntity).when(httpResponse)
                            .getEntity();

        final CloseableHttpClient httpClient = mock(CloseableHttpClient.class);
        final HttpUriRequest httpGet = mock(HttpUriRequest.class);
        doReturn(httpResponse).when(httpClient)
                              .execute(httpGet);

        // mock sling
        final SlingHttpServletRequest request = mock(SlingHttpServletRequest.class);
        final SlingHttpServletResponse response = mock(SlingHttpServletResponse.class);
        final ServletOutputStream outputStream = mock(ServletOutputStream.class);
        doReturn(outputStream).when(response)
                              .getOutputStream();

        // execute with suppliers
        this.servlet.httpClient(() -> httpClient)
                    .httpRequest(value -> {
                        assertEquals(uri, value);
                        return httpGet;
                    })
                    .doGet(request, response);

        // verify headers were set
        verify(response).setContentType("application/json");
        verify(response).setCharacterEncoding("utf-8");
        // verify uri was retreived from config
        verify(this.configuration).uri();
        // verify the entity was written to the output stream
        verify(httpEntity).writeTo(outputStream);
    }
}
Static mocking is gone, replaced by the functional interfaces. They will provide the mock instances of HttpGet and HttpClient. I am happier with this, yet the unit test is still long and bound to become a maintenance issue.

In-Memory HTTP Server

A couple of years ago, we had the same problem with mocking the Sling & JCR APIs. It was hard! Resource resolvers and sessions. Resources and nodes. Properties and value maps. Then, along came Sling Mock APIs & wcm.io AEM Mocks. This gave us an in-memory AEM environment suitable for testing. Wouldn’t it be nice if there was an in-memory HTTP server?

There are lots actually. To name a few:

I’m not here to compare and contrast the different implementations. That would be up to you and the set of features you think you need. For my example, I am happy with Mock Web Server primarily because it is lightweight and has a simple queuing mechanism. Let us see how fast I can mock this up.

@ExtendWith({ MockitoExtension.class, AemContextExtension.class })
class MockServerProxyServletTest {

    @Mock
    private ProxyServlet.Configuration configuration;
    @InjectMocks
    private ProxyServlet servlet;

    private MockWebServer mockWebServer;

    @BeforeEach
    void beforeEach() throws IOException {

        this.mockWebServer = new MockWebServer();
        this.mockWebServer.start();
    }

    @AfterEach
    void afterEach() throws IOException {

        this.mockWebServer.shutdown();
    }

    @Test
    void doTest(final AemContext context) throws IOException {

        // setup
        final String url = this.mockWebServer.url("/").toString();
        doReturn(url).when(this.configuration).uri();
        this.mockWebServer.enqueue(new MockResponse().setBody("hello, world!"));

        // execute
        this.servlet.doGet(context.request(), context.response());

        // assert
        final String result = context.response().getOutputAsString();
        assertEquals("hello, world!", result);
    }
}

In the end, I wind up using all three testing frameworks. Mockito to mock the OSGi configuration. AEM Mocks to mock the Sling HTTP serverlet request/response. Last, Mock Web Server to mock the remote REST API. It is all so much simpler. You no longer have to worry about managing so many mocks. In fact, your unit test is now client agnostic. Even if you go back to your proxy and switch out the Apache HTTP Client for the Java 11 HTTP Client, it will still work without a rewrite to the unit test.

Conclusion

AEM’s OSGi container is not the same as Spring’s IoC Container. Static and construction mocking is not an anti-pattern. It should definitely not be your first choice. Instead, favor good design patterns until the cost outweighs the benefits. Then, you can fall back to Mockito’s static and construction mocking, if not PowerMock’s.

Be sure to leverage a good mocking framework. Mockito provides basic mocking. Also, wcm.io provides a framework that will cut down on mocking, making tests maintainable. Likewise, a good HTTP mocking server will do the same for your HTTP client code, regardless of what HTTP client you use.

Note On Dependencies

In order to make the examples above work, I used an AEM Maven Archetype Project

mvn -B archetype:generate \
    -D archetypeGroupId=com.adobe.aem \
    -D archetypeArtifactId=aem-project-archetype \
    -D archetypeVersion=26 \
    -D appTitle="My Site" \
    -D appId="mysite" \
    -D groupId="com.mysite" \
    -D aemVersion=6.5.0

Make the following changes to pom.xml

<!-- 1. update version of junit-bom -->
<dependency>
    <groupId>org.junit</groupId>
    <artifactId>junit-bom</artifactId>
    <!--<version>5.6.2</version>-->
    <version>5.7.0</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

<!-- 2. replace mockito-core with mockito-inline -->
<dependency>
    <groupId>org.mockito</groupId>
    <!--<artifactId>mockito-core</artifactId>
    <version>3.3.3</version>-->
    <artifactId>mockito-inline</artifactId>
    <version>3.11.2</version>
    <scope>test</scope>
</dependency>

Make the following changes to core/pom.xml

<!-- 1. add the following two -->
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.20</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>com.squareup.okhttp3</groupId>
    <artifactId>mockwebserver</artifactId>
    <version>4.9.1</version>
    <scope>test</scope>
</dependency>

<!-- 2. remove junit-addons -->
<!-- <dependency>
    <groupId>junit-addons</groupId>
    <artifactId>junit-addons</artifactId>
    <scope>test</scope>
</dependency>-->

<!-- 3. replace mockito-core with mockito-inline -->
<dependency>
    <groupId>org.mockito</groupId>
    <!--<artifactId>mockito-core</artifactId>-->
    <artifactId>mockito-inline</artifactId>
    <scope>test</scope>
</dependency>

About the Author

Juan Ayala is a Lead Developer in the Adobe practice at Perficient, Inc., focused on the Adobe Experience platform and the things revolving around it.

More from this Author

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up