Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained […] infancy is perpetual. Those who cannot remember the past are condemned to repeat it. George Santayana

To understand where we are going in Web Content Management, we must first understand how we got to where we are now. Let’s embark on a semi-accurate and mildly satirical journey through the history of Web Content Management.

In the Beginning…

In the beginning, the web developer created raw HTML without form or process and saw it was bad.

Incantations such as FrontPage and Dreamweaver could produce elegant pages from arcane HTML tags, but code and content were inexorably mixed any changes to words or images required developer support.

Developers grew tired of marketing buzzing on about customer journeys, content velocity, and interactions and decided something must be done to provide separation between content and code.

The Expanse of XML

And the web developer said, “Let there be XML in the breadth of the codebase to distinguish between the content and the code and let the XML be called content and have a system for marketers to manage it so I don’t have to”. Along came enterprise software vendors with solutions to convert large piles of cash into maintainable websites.

The web developer wrote code in XSLT and marketers wrote content in XML and they saw it was bad.

Content and code were theoretically separate, but new features and changes still required coordination between marketing and web developers and interactive web applications and marketing websites were divided by a chasm deeper than the darkest trenches of the seas.

Rise of the Web Content Frameworks

Lamenting the disconnect between web apps and websites, the web developer said “if only I could have one system that would join a content management system and web application framework as one, then it would finally be good”. And came Drupal, Adobe Experience Manager, and Sitecore which provided all of the features the marketer and web developer wanted out of the box. 

Lamentably, the marketer had eaten from the tree of design and saw that the website was naked. The web developer extended the out of the box features and soon saw it was bad.

Upgrades were excruciating and due to the entanglement of custom and framework code, seemingly small changes required many hours and significant costs.

Single Page Everything

Away in the mystic land of Silicon Valley, the one true prophet of technology, introduced the one true Single Page Application framework React. Shortly thereafter, the other one true prophet of technology introduced Angular which is also the one true framework for Single Page Applications. 

These Single Page Applications enabled the web developer to create websites that avoid the one thing that bothers users more than anything else, page reloads and so the marketer and web developer decided to rebuild all of their sites so they would finally be good.

The web developer rebuilt the site’s Single Page Apps and the marketer saw it was bad. No longer self-enabled, every change once again had to through a development release process. 

Let Them GET /cake.json

Storming the bastille of the now-traditional CMS paradigm, a new cadre of headless CMS solutions fomented a revolution to overthrow the class system of static websites and dynamic CMS systems and replace it with an egalitarian, universal Content API.

Swept up in revolutionary fever, the web developer and the marketer again re-implemented half of the website on a new platform before realizing it was also bad. 

No longer able to leverage a base framework, the web developer had to reimplement the wheel, while the marketer struggled to understand the context of the content without visualizing it on a page with the form-based content authoring. Thus the cycle continued and the universe collapsed on itself in an explosion of budgets and technology debt when the web developer mentioned re-implementing with GraphQL.

Learning from the Past

Web Content Management is a discipline which takes a week to learn and a career to master. While at the simplest, it is just putting markup and binary assets on the internet, the inherent contradictions in needs, goals, and capabilities introduce tremendous challenges.

Based on my experience in this industry and many lessons learned I’ve come to the following conclusions:

• One-size-fits-all solutions are rarely completely right, most often the solution involves multiple approaches working together
• If choosing a one-size-fits-all solution, you must be clear on what you are giving up and ensure those compromises are worth the architectural simplification
• When defining the content structure, start with Authoring, not the experience being created. If you understand how content will go into the system and then how it will be exposed, you will understand the optimal approach to content
• The interrelation of code and content in HTML requires balancing the desire to create rich content with the technical debt, complexity, and brand consistency challenges this introduces

What’s your takeaway? Leave a comment below and let’s discuss!

Recently, I needed to perform a query using the AEM Query Builder which was case insensitive. While I normally prefer using JCR SQL2 queries, in this case Query Builder was a better fit as I wanted consuming applications to be able to manipulate the query and doing so as a map is significantly easier than doing so as a string.

I was surprised to find that there was no native Query Builder Predicate for doing case insensitive queries so I ended up writing my own.

The predicate works by lower casing the value and then using the XPath fn:lower-case method to compare to the field value in lower case.

/*
This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to 
*/
package com.perficient.adobe.predicates;

import java.util.Locale;
import java.util.Optional;

import com.day.cq.search.Predicate;
import com.day.cq.search.eval.AbstractPredicateEvaluator;
import com.day.cq.search.eval.EvaluationContext;

import org.osgi.service.component.annotations.Component;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component(factory = "com.day.cq.search.eval.PredicateEvaluator/equalsIgnoreCase")
public class CaseInsensitiveEquals extends AbstractPredicateEvaluator {

    private static final Logger log = LoggerFactory.getLogger(CaseInsensitiveEquals.class);

    static final String PREDICATE_PROPERTY = "property";
    static final String PREDICATE_VALUE = "value";
    static final String PREDICATE_LOCALE = "locale";

    @Override
    public String getXPathExpression(Predicate predicate, EvaluationContext context) {
        log.debug("Evaluating predicate: {}", predicate);
        String property = predicate.get(PREDICATE_PROPERTY);
        Locale locale = Optional.ofNullable(predicate.get(PREDICATE_LOCALE)).map(lt -> Locale.forLanguageTag(lt))
                .orElse(Locale.getDefault());
        String value = predicate.get(PREDICATE_VALUE).toLowerCase(locale).replace("'", "''");
        String query = String.format("fn:lower-case(@%s)='%s'", property, value);
        log.debug("Generated query: {}", query);
        return query;
    }
} 

Once the custom predicate is available in your application, it can be used in any Query Builder query as such:

path=/content/test
equalsIgnoreCase.property=test
equalsIgnoreCase.value=test
equalsIgnoreCase.locale=en_US 

The locale parameter is not required, but would generally be recommended unless the input will always be in the system default locale.

In the first post of the Exploring the Sling Feature Model series, I discussed the process of converting the Sling CMS app from the Sling Provisioning Model to the Sling Feature Model. So how does this apply to your custom applications?

To illustrate, let’s convert my personal site, danklco.com, which is currently managed via Sling CMS, to the Feature Model.

It’s worth noting that I could keep running my site the way it is, by using a pre-built Sling CMS Runnable Jar, but that my goal is to run my site in Kubernetes for simplicity of upgrades, deployment, and management.

Step 1: Refactor Project Structure

Currently, my personal website code is a single OSGi Bundle which I deploy with Github Actions. To support the Sling Feature Model, I’m going to convert the project into a multi-module project and add a new sub-project for my feature.

The new project structure will look like:

/mysite
/bundle
/feature
/images

Step 2: Generate Features

The custom feature is pretty simple, defining my custom code bundles and configurations. A number of parameters are supplied so they can be changed and I’m not putting secrets in code:

{
    "bundles": [
        {
            "id": "com.danklco:com.danklco.slingcms.plugins.disqus:1.1-SNAPSHOT",
            "start-order": "20"
        },
        {
            "id": "com.danklco:com.danklco.slingcms.plugins.twitter:1.0",
            "start-order": "20"
        },
        {
            "id": "com.danklco:com.danklco.site.cna.bundle:1.0.0-SNAPSHOT",
            "start-order": "20"
        }
    ],
    "configurations": {
        "org.apache.sling.cms.core.analytics.impl.GeoLocatorImpl": {
            "scheduler.expression": "0 0 0 ? * WED",
            "licenseKey": "${MAXMIND_LICENSE_KEY}"
        },
        "org.apache.sling.cms.reference.impl.SearchServiceImpl": {
            "searchServiceUsername": "dklco-com-search-user"
        },
        "org.apache.sling.commons.crypto.internal.FilePasswordProvider~default": {
            "names": [
                "default"
            ],
            "path": "/opt/slingcms/passwd"
        },
        "org.apache.sling.commons.crypto.jasypt.internal.JasyptRandomIvGeneratorRegistrar~default": {
            "algorithm": "SHA1PRNG"
        },
        "org.apache.sling.commons.crypto.jasypt.internal.JasyptRandomSaltGeneratorRegistrar~default": {
            "algorithm": "SHA1PRNG"
        },
        "org.apache.sling.commons.crypto.jasypt.internal.JasyptStandardPBEStringCryptoService~default": {
            "algorithm": "PBEWITHHMACSHA512ANDAES_256",
            "saltGenerator.target": "",
            "securityProviderName": "",
            "ivGenerator.target": "",
            "securityProvider.target": "",
            "keyObtentionIterations": 1000,
            "names": [
                "default"
            ],
            "stringOutputType": "base64"
        },
        "org.apache.sling.commons.messaging.mail.internal.SimpleMailService~default": {
            "connectionListeners.target": "",
            "transportListeners.target": "",
            "username": "${SMTP_USERNAME}",
            "mail.smtps.from": "${SMTP_USERNAME}",
            "messageIdProvider.target": "",
            "mail.smtps.host": "${SMTP_HOST}",
            "names": [
                "default"
            ],
            "password": "${SMTP_ENC_PASSWORD}",
            "mail.smtps.port": 465,
            "cryptoService.target": "",
            "threadpool.name": "default"
        },
        "org.apache.sling.commons.messaging.mail.internal.SimpleMessageIdProvider~default": {
            "host": "danklco.com",
            "names": [
                "default"
            ]
        }
    }
}

To create a usable model, I’ll need to combine the Sling CMS model and my custom model, which can be accomplished with the Sling Feature model. To support the Composite Node Store, I’ll want to generate two separate aggregates, one for seeding and one for running the instance.

Since the Sling Feature Model JSON will resolve dependencies at runtime from Apache Maven, we’ll also want to generate Feature Archives or FAR files which bundles the models with their dependencies.

<plugin>
    <groupid>org.apache.sling</groupid>
    <artifactid>slingfeature-maven-plugin</artifactid>
    <version>1.3.0</version>
    <extensions>true</extensions>
    <configuration>
        <framework>
            <groupid>org.apache.felix</groupid>
            <artifactid>org.apache.felix.framework</artifactid>
            <version>6.0.3</version>
        </framework>
        <aggregates>
            <aggregate>
                <classifier>danklco-com-seed</classifier>
                <filesinclude>**/*.json</filesinclude>
                <includeartifact>
                    <groupid>org.apache.sling</groupid>
                    <artifactid>org.apache.sling.cms.feature</artifactid>
                    <version>0.16.3-SNAPSHOT</version>
                    <classifier>slingcms-composite-seed</classifier>
                    <type>slingosgifeature</type>
                </includeartifact>
                <includeartifact>
                    <groupid>org.apache.sling</groupid>
                    <artifactid>org.apache.sling.cms.feature</artifactid>
                    <version>0.16.3-SNAPSHOT</version>
                    <classifier>standalone</classifier>
                    <type>slingosgifeature</type>
                </includeartifact>
                <title>DanKlco.com</title>
            </aggregate>
            <aggregate>
                <classifier>danklco-com-runtime</classifier>
                <filesinclude>**/*.json</filesinclude>
                <includeartifact>
                    <groupid>org.apache.sling</groupid>
                    <artifactid>org.apache.sling.cms.feature</artifactid>
                    <version>0.16.3-SNAPSHOT</version>
                    <classifier>slingcms-composite-runtime</classifier>
                    <type>slingosgifeature</type>
                </includeartifact>
                <includeartifact>
                    <groupid>org.apache.sling</groupid>
                    <artifactid>org.apache.sling.cms.feature</artifactid>
                    <version>0.16.3-SNAPSHOT</version>
                    <classifier>standalone</classifier>
                    <type>slingosgifeature</type>
                </includeartifact>
                <title>DanKlco.com</title>
            </aggregate>
        </aggregates>
        <scans>
            <scan>
                <includeclassifier>danklco-com-seed</includeclassifier>
            </scan>
            <scan>
                <includeclassifier>danklco-com-runtime</includeclassifier>
            </scan>
        </scans>
        <archives>
            <archive>
                <classifier>danklco-com-seed-far</classifier>
                <includeclassifier>danklco-com-seed</includeclassifier>
            </archive>
            <archive>
                <classifier>danklco-com-runtime-far</classifier>
                <includeclassifier>danklco-com-runtime</includeclassifier>
            </archive>
        </archives>
    </configuration>
    <executions>
        <execution>
            <id>aggregate-features</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>aggregate-features</goal>
                <goal>analyse-features</goal>
                <goal>attach-features</goal>
                <goal>attach-featurearchives</goal>
            </goals>
            <configuration>
                <replacepropertyvariables>MAXMIND_LICENSE_KEY,SMTP_HOST,SMTP_USERNAME,SMTP_ENC_PASSWORD</replacepropertyvariables>
            </configuration>
        </execution>
    </executions>
</plugin>

Step 3: Build Docker Images

Since the goal is to run this in Kubernetes, we’ll create Docker images for running Sling CMS and Apache web server. Since I’m running a lean server, I’ll want to run this as a standalone instance using the Composite Repository so the datastore persists between instances.

To populate variables into the images and coordinate the full build, we’ll use Apache Maven to process the Docker files and input files as Maven artifacts and kick off the docker build. Unlike the Sling CMS build, we’re not leveraging Apache Maven to download the artifacts within the Docker build, we’ll pre-fetch them during the maven build and supply them to the Docker build.

Side Note – Variables

One challenge to note when attempting to reproduce an actual instance, there are a quite a few variables required for the application to actually work. For my local testing I have a bash script to provide all of the required properties to Maven, but since they include secrets like passwords I’ve not put it in source control.

See it in Action!

Seeing something work is work a thousand words, so check out this GIF of the build process in action:

and check out the code on GitHub: https://github.com/klcodanr/danklco.com-site/tree/cloud-native-sling

What’s Next?

All of this is leading up to having a fully running Cloud Native Apache Sling CMS instance in Kubernetes, but before that my next post is going to talk about using Sling Content Distribution and Sling Discovery to support publishing content between Author and Renderer Apache Sling CMS instances. Check back soon!

In a comment on my previous post Apache Sling JVM Performance, Gil Tene made an insightful comment about the potential possibility of performance impact from speed from the underlying environment or other tests.

To accommodate for this possibility, I re-ran the tests inside a loop, randomizing order of the JVM execution for each iteration 25 times. As Gil predicted this did bring the OpenJDK implementations closer together with GraalVM and OpenJ9 as outliers. 

Startup Performance

Interestingly, with the multiple iterations, OpenJ9 actually became the fastest starting JVM implementation, though practically tied with OpenJDK Hotspot and Azul Zulu. GraalVM was almost 6 seconds slower to start on average.

Package Installation Performance

Package performance was quite interesting as every JVM besides OpenJ9 averaged out nearly identically. 

Transaction Performance

The transaction performance varies significantly from the initial results with GraalVM taking the lead in the rate and quantity of transactions and OpenJ9 handles almost 5 transactions fewer per second than GraalVM.

This is honestly quite different than I expected. My hypothesis was that the OpenJDK-based implementations would net out pretty similarly, but in actuality, there was a statistically significant difference between each implementation.

Memory Usage

The full run of 25 iterations showed roughly the same results in terms of memory usage. OpenJ9 used significantly less memory and GraalVM significantly more with OpenJ9 using 60% of the average memory of the OpenJDK implementations.

Outliers

One of the interesting things to observe is that there were some extreme outliers, for example, package installation which generally took ~30 seconds, occasionally taking over 2.5 minutes. It seems like this is related to the underlying hardware as there’s not a pattern in the iteration order, iteration number of JVM implementation. To avoid skewing the data, I excluded these outliers from the other charts.

Revised Summary and Findings

With multiple runs, the differences between the OpenJDK codebase implementations (e.g. OracleJDK, Amazon Coretto, Azul Zulu and OpenJDK Hotspot), reduces significantly. The performance and startup differences small enough, that licensing would be the primary criteria I’d recommend when considering the JVM implementation to use.

If raw performance is the primary concern, GraalVM demonstrated a consistently higher transaction rate over the iterations at the cost of a slower startup and higher memory usage.

For lower-end usages or container-based usages, OpenJ9 continues to be an excellent choice with it’s low memory usage and especially after it demonstrated the promised faster startup on average over the multiple iterations. 

With the recent proliferation of Java Virtual Machine (JVM) implementations, it’s difficult to know which implementation is the best for your use case. While proprietary vendors generally prefer Oracle Java, there are several open source options with different approaches and capabilities. 

Given how the implementations vary in some underlying technical specifics, the “correct” JVM implementation will vary based on the use case. Apache Sling, in particular has some specific needs given the complexity of the OSGi / JCR backend and the Sling Resource Resolution framework. 

Test Strategy

To help get some real data on which JVM implementation works best for running Apache Sling, I created a project to:

1. Install a number of JVM implementations and monitoring tools
2. For each JVM:
   1. Setup an instance of Apache Sling CMS, using no additional parameters
   2. Install a content package to the Sling CMS instance
   3. Run a performance test using siege
3. Consolidate the data into a CSV

If you are curious, you can checkout the Sling JVM Comparison project on Github.

The project installs and compares the following JVM implementations on version 11:

• Eclipse OpenJ9 – 11.0.8 – openj9-0.21.0, JRE 11
• Amazon Coretto – 11.0.8.10.1
• OpenJDK Hotspot – 11.0.8+10
• Oracle JDK – 11.0.8+10-LTS
• Azul Zulu – 11.0.8+10-LTS – 11.41+23-CA
• GraalVM – CE 20.1.0 – 11.0.7+10-jvmci-20.1-b02

To create a meaningful comparison, I setup and ran the test an Amazon EC2 m5.large instance running Ubuntu 18.04 LTS “Bionic Beaver” and captured the results.

Startup / Install Performance

An important performance comparison is the amount of time it takes to get an instance running. To measure this, I captured the time in milliseconds to start the Apache Sling CMS instance and the amount of time required to upload and install the same content package. There is a potential variance in the capture of the startup time as the test process polls the Sling instance to see when it responds successfully to a request to determine startup time.

OpenJDK Hotspot and Amazon Coretto are essentially tied as the leaders of the pack with Oracle JDK and GraalVM following shortly behind. Azul Zulu and Eclipse OpenJ9 take 78% and 87% longer to start as OpenJDK Hotspot. Interestingly, most of the JVM implementations take approximately the same time to install the content package, however, Eclipse OpenJ9 takes 35% longer to install the content package.

Performance Under Load

To check performance under load, I tested the instances using siege using a list of URLs over the course of an hour with blocks of 15 minutes on and 15 minutes off. 

First, we can take a look at the throughput per second:

And next, we can look at the raw transaction count:

Both show the same story, OpenJDK Hotspot, Amazon Coretto and Oracle JDK taking the top spots for performance with GraalVM, Azul Zulu and Eclipse OpenJ9 trailing behind.

Memory Usage

Finally, given how memory intensive Java applications can be, it’s important to consider memory usage and here the differences are quite stark:

Eclipse OpenJ9 is significantly less memory intensive, using only 55% of the average memory of the 4 middle-tier JVM implementations. GraalVM also sits outside the average, using 15% more memory than the same middle-tier JVM implementations.

Summary and Findings

From a raw performance perspective, OpenJDK Hotspot is the clear winner with Amazon Coretto close behind. If you are all in on Amazon or want a long-term supported JVM option, Amazon Coretto would be worth considering.

For those running Apache Sling on memory-limited hosting options, Eclipse OpenJ9 is the best option. While there is a tradeoff for performance, when you only have a Gigabyte or two of memory, reducing the load by 45% will make a tremendous difference.

Credit

Thanks to Paul Bjorkstrand for coming up with idea for this post. 

Working with Adobe Managed Services (AMS) I’ve wished I had a guide to the common activities I need to log into a server to perform. Here is a quick starter to help you find your way around the servers provisioned by Adobe Managed Services. Since I can’t guarantee AMS environments are consistent (and checking with other teams I’ve confirmed they’re not) you may find these commands or paths don’t exactly match yours, but they should be a good start.

SSH Access

Before you can SSH into the AMS hosts, you will need to reach out to the CSE in order to create a user name and add your public key to the ~/.ssh/authorized_keys file on the host. I’d note that some AMS customers are still issued password-based users (in 2020!!) so ssh-copy-id may come in handy.

By default, the CSE will create a single username for all users to access. You can request they create named users as well, however, you will need to be vigilant in adding/removing users as these users are not managed in a centralized repository.

Lower vs. Upper Environment Access

By default, you will have more (if still limited) access to the Lower environments as compared to the upper environments. Even in the upper environments, you can request a “jailed user” with read-only access to the logs.

In the lower environments, you will be able to do a number of “write” activities by invoking sudo. For a full list of the commands you can execute run:

sudo -l

Your allowed sudo commands will generally use the full file path, it’s important to note in that case, you cannot just execute the command from the relative directory, e.g. this won’t work:

cd /etc/httpd/conf.d
sudo vi dispatcher_vhost.conf

but this will:

sudo vi /etc/httpd/conf.d/dispatcher_vhost.conf

Dispatchers

For those not experienced with AEM, Dispatcher servers run Apache httpd with a special module called the Dispatcher and serve as a proxy, cache and quasi-security layer for the AEM Authors and Publishers.

Important Directories:

• Logs: /mnt/var/log/httpd/
  Note – you will not be able to change to the log directory and must use sudo
• Apache Docroot: /mnt/var/www/html/
• Apache Server Configuration: /etc/httpd/
• Dispatcher Configuration: /etc/httpd/conf.dispatcher.d/

Useful Commands:

• Restart Apache: sudo service httpd restart
• Diagnose build errors: sudo journalctl --system -u httpd
• List the Apache environment variables: cat /etc/sysconfig/httpd
• List log files: sudo ls /mnt/var/log/httpd/
• Tail log file: sudo tail -f /mnt/var/log/httpd/[log-file-name]
  Note – you cannot tail /mnt/var/log/httpd/*, you need to explicitly mention the log files to tail, e.g:
  sudo tail -f /mnt/var/log/httpd/access_log /mnt/var/log/httpd/error_log
• Edit a dispatcher configuration file: sudo vi /etc/httpd/conf.dispatcher.d/[configuration-file]

AEM Instances

This applies for both AEM Author and Publish instances as the setup of each from the AMS perspective is nearly identical besides the runmode.

Directories:

• AEM Installation: /mnt/crx/[author|publish]/crx-quickstart
  Note – You won’t be able to access the parent of the crx-quickstart folder and have to change directly into the crx-quickstart path
• Logs: /mnt/crx/[author|publish]/crx-quickstart/logs

Useful Commands:

• Restart AEM: sudo /etc/init.d/cq5 restart
• List Logs: ls /mnt/crx/[author|publish]/crx-quickstart/logs
• Tail logs: tail -f /mnt/crx/[author|publish]/crx-quickstart/logs/[log-file]

Hopefully, this helps you feel your away around your AMS installation and remember, we’re always here to help. Having trouble getting to something or diagnosing an issue? Leave a comment!

In my previous post Exploring the Sling Feature Model, I described the process of migrating from a Sling Provisioning project setup to a Sling Feature Model project. 

Now that we have the Sling Provisioning Model project converted, we can move on to the fun stuff and create a Composite NodeStore. We’ll use Docker to build the Composite Node Store in the container image.

Creating the Composite NodeStore Seed

The Composite NodeStore works by combining one or more static “secondary” node stores with a mutable primary NodeStore. In the case of AEM as a Cloud Service, the /apps and /libs directories are mounted as a secondary SegmentStore, while the remainder of the repository is mounted as a MongoDB-backed Document Store. 

For our simplified example, we will create a secondary static SegmentStore for /apps and /libs and combine that with a primary SegmentStore for the remainder of the repository. Since the secondary SegmentStore will be read-only, we must “seed” the repository to pre-create the static paths /apps and /libs. 

To do this, we have a feature specifically to seed the repository with the /apps and /libs temporarily mutable. We can then use the aggregate-features goal of the Sling Feature Maven Plugin to combine this with the primary Feature Model to create a feature slingcms-composite-seed. When we start a Sling instance using this feature, it will create the nodes under these paths based on the feature contents. 

As shown below, while seeding the repository is written to the libs SegmentStore. It’s also worth mentioning that with the Feature Model Launcher, by default, the OSGi Framework runs in a completely different directory from the repository and pulls the bundle JARs from the local Maven repository.

Our updated Dockerfile runs the following steps to build the container image:

• Downloads the Feature Model Launcher JAR and Feature Model JSON files
• Starts the Sling instance using the slingcms-composite-seed model in the background
• Polls the Felix Health Checks until the tag “systemalive” returns 200
• Once the 200 status is returned the Sling instance is stopped and the build cleans up the launcher and symlinks the SegmentStore directory into the expected path

Naming Gaffes

Hopefully, you are more careful than me, but one thing to keep in mind is that the Sling Feature Launcher will happily start as long as it has a valid model. For example, you can easily spend a significant amount of time trying to understand why nothing responds with this model:

org.apache.sling.cms.feature-0.16.3-SNAPSHOT-composite-seed.slingosgifeature

Instead of the one I meant:

org.apache.sling.cms.feature-0.16.3-SNAPSHOT-slingcms-composite-seed.slingosgifeature

Since the non-aggregate model is a valid model, the Sling Feature Launcher will happily start, but it simply creates an OSGi container with only a couple of configuration which naturally does… nothing. 

Starting and Running

Once the repository is been fully started and seeded, we’ll run a different Feature Model to run the instance. Similar to the Composite Seed Feature Model, the slingcms-composite-runtime Composite Model will use the composite repository, however it runs the libs mount in readonly mode.

To use the runtime Feature Model, the CMD directive in the Dockerfile calls the Sling Feature Model Launcher with the slingcms-composite-runtime Feature Model. In addition, we’ll mount a volume in the docker-compose.yml to separate the mutable volume out from the container disk, that way the repository persists between restarts and container deletion.

While in runtime mode, the Composite repository looks like the diagram below, leveraging a Docker volume for the global SegmentStore and the local seeded repository for the libs SegmentStore:

End to End

Here’s a quick video showing the process of creating a Container-ized version of Sling CMS with a Composite NodeStore from end to end.

Details: Build Arguments & Dependencies

The current example implementation uses Apache Maven to pull down the Feature Models with a custom settings.xml and Build Arguments in the Dockerfile. By changing the settings.xml and the Build Arguments, you could override the Feature Model being produced to use a custom Feature Model, for example an aggregate of Sling CMS and your custom Sling CMS app.

We’ll cover the process of producing a custom aggregate in the next blog post in the Exploring the Sling Feature Model series. If you’d like to learn more about the Sling Feature Model, you should check out my previous post on Converting Provisioning Models to Feature Models.

In my previous post about AEM as a Cloud Service, How AEM Scales, I discussed the new Sling Feature Model which is a key part of how Adobe scales their new AEM as a Cloud Service solution using Kubernetes. 

The goal of the Feature Model is to create a better way to provision Sling instances by creating a more descriptive, flexible grammar for building targeted instances instead of building a fat JAR file and adding bundles after the fact.

This week / weekend, I took some time to finally upgrade my pet project, Sling CMS to use the Sling Feature Model and wanted to take a moment to document the process and my learnings.

Once you understand the process and model, the process of converting a Provisioning Model-based project to Feature Model is straight-forward. To get started, I followed along with the Feature Model How-To Guide and then dug into the code for the Sling Kickstart and (Feature Model compatible branch of) Sling Starter. 

Converting a Provisioning Model Project to Feature Model

The first step was using the sling-feature-converter-maven-plugin to convert the legacy Provisioning Model configuration to the Feature Model. To do this, I created a temporary sub-module to execute the conversion process:

https://github.com/apache/sling-org-apache-sling-app-cms/blob/SLING-8913-multiple-instance-types/converter/pom.xml

I’ve modified this POM based on the Sling Kickstart POM to look up the Provisioning files in the builder sub-module of Sling CMS, configured it with my desired artifact information and added a step to create an aggregated Feature Model so I could validate the conversion.

Interestingly, I found that mvn clean after failed builds seemed to often result in Maven dependency resolution issues (I’m assuming because it was evaluating the Feature files) so rm -rf target/ was my go to cleaning action.

To run the conversion, I can execute mvn clean install -Dbnd.baseline.skip=true. Once the process is complete, the feature files can be found under target/fm and you can validate by downloading the Feature Model Launcher and running:

java -jar org.apache.sling.feature.launcher-1.1.4.jar -f target/slingfeature-tmp/feature-slingcms.json

One feature file will be created per Provisioning Model file and the contents of the Provisioning Model file will be transformed into the format used by the Feature Model.

During this step, I also found that the Sling Feature Model is more strict than the Provisioning Model and called out a few mistakes in the Provisioning Model, including duplicate configurations (for example I had duplicated the configuration org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.amended-sling.rewriter).

Once I successfully converted the Provisioning files to Feature Models, I copied them from target/fm into a new features submodule and updated the Artifact ID accordingly. 

Aggregating the Feature Model

The purpose of the new feature submodule is to produce the final artifacts based on the Feature Model files. The first and easiest artifact is to create an aggregate feature to execute with the Sling Feature Model Launcher. 

To do this, I added the following goals from the slingfeature-maven-plugin plugin:

• aggregate-features – aggregate the feature files into a single feature file
• analyse-features – analyse the feature files to ensure they didn’t have any issues
• attach-features – attach the feature models as Maven artifacts
• attach-featurearchives – attach the feature archives as Maven artifacts

The outcome of all this is a consolidated Feature Model and Feature Archive for Sling CMS based on the input Feature Model files we migrated from the Provisioning Model. Before we can complete the process however, there are a few issues which need to be resolved. 

Feature Models vs. Feature Archives

Quick segue about Feature Archives. By default the Sling Feature Model will download dependencies as artifacts from the Maven repository. To avoid having to download files at runtime, package private artifacts or to enable Sling Features to run offline Sling Feature Model supports Feature ARchives or FAR files. The FAR file packages the Feature descriptor and binary dependencies into a ZIP archive.

Feature Archives can be used in the same manner as Feature Models, reference the path to the Feature Archive instead of the Feature Model in the -f or -sf flags for the Sling Feature Launcher.

Resolving Feature Errors

The analyse-features goal of the slingfeature-maven-plugin flagged a number of issues with the legacy Sling CMS Provisioning Model setup around the start order of the bundles. Each error will be logged with a message like the following:

[ERROR] Artifact org.apache.sling:org.apache.sling.servlets.resolver:2.7.2 requires [org.apache.sling.servlets.resolver/2.7.2] osgi.ee; filter:="(&(osgi.ee=JavaSE)(version=1.8))" in start level 20 but no artifact is providing a matching capability in this start level.

Blindly updating start order to resolve the build issues caused Sling to not be able to render content. I would recommend making changes one bundle (or set of bundles) at a time and validating with every change. In the Sling CMS setup, the primary problem I had was with the ordering of the Apache Sling integration with Apache Oak and the bundles org.apache.sling.jcr.jackrabbit.usermanager and org.apache.sling.jcr.jackrabbit.accessmanager.

Version Variables

One of the conveniences in the Provisioning model is Variables, which allow you to set shared variables for things like version numbers when you have multiple bundles from the same project (Oak, Jackson, Composum, etc) you want to keep in sync. The Feature Model has a similar concept called Placeholders.

The great thing about Placeholders is they can be injected from the POM. For example, I created the following properties in my POM’s properties elements:

<composum.version>1.12.0</composum.version>
<jackrabbit.version>2.20.0</jackrabbit.version>
<jackson.version>2.11.1</jackson.version>
<oak.version>1.26.0</oak.version>
<slf4j.version>1.7.25</slf4j.version>

Each instance of these variables will be replaced within all of the Feature Model files.

Creating a Runnable JAR

While the goal of the Feature Model is to create a more flexible model and get away from running a fat JAR, but for local development and getting started quickly having a runnable JAR is just easier. 

To create a runnable JAR as a part of the same build, I configured Maven to combine the Sling Feature Model Launcher and my Sling CMS Feature Archive at build time into a standalone JAR using the Maven Assembly Plugin.

The Main class / method is quite brief, it:

• Gets the version number passed via a properties file
• Resolves the URL for the Feature Archive from the classpath
• Appends (if not present) the -f flag with URL of the Feature Archive to the arguments
• Calls the Sling Feature Model Launcher Main method

With that, we now have a standalone JAR which provides a double click run of Sling CMS as well as the rich Feature Model based aggregation in the form of Feature Models and Feature Archives all in one build.

In the next blog post in the Exploring the Feature Model series, I will be setting up an Oak Composite Node Store. 

Adobe’s Cloud Manager is quite feature rich in unexpected features. One of these features is that it is in no way obvious how one can diagnose dispatcher deployment issues when run through Cloud Manager.

Note: due to AMS’ inconsistent environment setup, this article may not apply to everyone, but if you rely on your CSE to debug dispatcher deployment issues, this probably does for you. 

Why did my Cloud Manager Build Fail to Deploy Dispatcher?

The problem is that to test your deployment, Cloud Manager will unpack the deployment on the instances and restart to validate that the configuration is correct. If this fails, it rolls back the changes and logs a single message to the Cloud Manager log.

2020-07-07T17:15:40+0000 Failed to deploy dispatcher on instance dispatcher1eastus. ActionId=[some-action-id]

Based on that you’re clear on what you need to do? Of course not! Normally, I’d expect the details of restart / issues to be written to a log accessible via the build tool, but Adobe Managed Services in their endless desire to defy normal expectations choose not to do this. 

Apparently, Adobe Engineering is considering on a fix for this, but have not specified a timeframe or committed to the fix. At this point, you could reach out to your Customer Success Engineer, but really?? Isn’t this something you should be able to self service?

Digging into the Systemd Journal

We know that Cloud Manager is using systemd to restart the Apache httpd web server powering the dispatcher, so theoretically the status will be written to the systemd journal. Unfortunately, as AMS will only provision non-privileged users, our user can only see their own journal, not the journal for the full system including Cloud Manager.

Seemingly a dead end. However, if you execute sudo -l, you will see a number of interesting things commands you can execute: 

Among these is:

(ALL) NOPASSWD: /bin/journalctl -* *

Note the format, we need to specify two parameters including a flag to call journalctl with sudo… No idea why and I would not expect AMS to set up every environment the same way, so your milage will vary.

Based on this information, we can execute the following command, which will pull the most recent systemd journal entries for httpd, including the reason why Apache httpd failed to start with your dispatcher changes:

sudo /bin/journalctl --system -u httpd

For example, if I missed a variable DISP_LOG_LEVEL, I would see something like:

[date] [instance] httpd[8753]: [Wed Jul 22 21:47:34.643291 2020] [core:warn] [pid 8753] AH00111: Config variable ${DISP_LOG_LEVEL} is not defined
[date] [instance] httpd[8753]: AH00526: Syntax error on line 21 of /etc/httpd/conf.d/dispatcher_vhost.conf:
[date] [instance] httpd[8753]: Invalid value for directive DispatcherLogLevel, expected integer
[date] [instance] systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
[date] [instance] kill[8755]: kill: cannot find process ""
[date] [instance] systemd[1]: httpd.service: control process exited, code=exited status=1
[date] [instance] systemd[1]: Failed to start The Apache HTTP Server.
[date] [instance] systemd[1]: Unit httpd.service entered failed state.
[date] [instance] systemd[1]: httpd.service failed.
[date] [instance] systemd[1]: Starting The Apache HTTP Server...
[date] [instance] systemd[1]: Started The Apache HTTP Server.

Note that we see the dispatcher starting, this is because Cloud Manager watches to see if Apache restarts as expected with your dispatcher changes and will roll back the build if it fails. 

Now that you can pull dispatcher deployment logs, you can diagnose dispatcher issues without waiting for hours for a CSE to respond to email.

Want to learn more about Cloud Manager?

You may want to check out my previous Cloud Manager post on pulling private dependencies in Cloud Manager.

For customer’s using Adobe Managed Services, Cloud Manager has become the de facto system for deploying code to their Adobe Experience Manager environments. Among Cloud Manager’s many deficiencies is the inability to fetch artifacts from private remote repositories.

The good news is there’s a workaround for internet-facing artifact repositories. Unfortunately, non-internet facing artifact repositories are not supported as Cloud Manager spins up Docker containers to run the actual build process and they do not support a pool of IPs or virtual networks for the containers. According to feedback from Adobe, the Cloud Manager team is waiting on a feature from Microsoft Azure to allow all Cloud Manager Containers to be assigned an IP address out of a defined range. Once this feature is added to Azure, AMS will be able to provide an IP range to allow access to the artifact server.

Why Would I Want to Do This?

I think it’s certainly worth asking: why not just follow Adobe’s model with Cloud Manager and put everything in one big Maven project?

There are a few good reasons you may not want to follow the Adobe pattern:

• You are a larger enterprise and have multiple teams working on multiple different projects in parallel and consolidating on a single project would make releases and merging very difficult to manage
• You have integrations or dependencies on upstream teams for things like service connections, UI frameworks or custom integrations
• You want to integrate additional checks into your code not supported in Adobe Cloud Manager, but don’t want to have to run a build twice

Connecting to a Private Artifact Repository

The solution involves two parts:

• Working with your CSE to configure Environment Variables
• Overriding the settings.xml in the project

Custom Environment Variables

First, Adobe has a feature where your CSE can request the provisioning of Custom Environment variables, this allows you to specify private variables such as the connection URL, username or password. This will require your CSE to open a support ticket with an internal team at Adobe, so you will simply need to provide the keys and values to the CSE to open the ticket.

According to an inside source, this may come as a feature in the API, which is currently available but not supported in Cloud Manager.
Edit: Custom variables are now available as Pipeline Variables which can be specified in the Adobe I/O CLI.

Setting Your Artifact Server

In order to resolve the dependencies from your private repository, Maven needs three configuration steps.

First, add the repository into the pom.xml, for example:

Next, you’ll need to override the settings.xml to do this, create a file in the root of the project called .mvn/maven.config with the contents:

-s ./settings.xml

This will instruct Maven to use a settings.xml file located in the project directory.

Finally, create a settings.xml and use the Custom Environment variables to set the username and password for the server (the ID must match the pom.xml) as well as providing the default adobe-public artifact repository:

Once all the configurations are in place, trigger a Cloud Manager build and you should see Cloud Manager connecting to your private artifact repository to download the dependencies for the project.

Want to learn more about Cloud Manager?

You may want to check out my latest Cloud Manager post on diagnosing Cloud Manager dispatcher deployment issues.

Did you miss my webinar with the Detroit Adobe Experience Meetup on Sling RepoInit? Sling RepoInit is a newer technology which enables projects to set up permissions, configurations and content via OSGi configurations in a purpose-built grammar.

Check out the presentation deck and recording to see how RepoInit could be useful on your project.

Watch the Recording

Curious about using Sling RepoInit? Want to learn more in depth about how Sling RepoInit can enable your AEM DevOps team to manage the initial repository state in code?

I’ll be leading a virtual discussion on Sling RepoInit with the Detroit AEM Meetup on Thursday July 9th from 6:00 – 6:50 PM EST.

This talk will:

• Introduce the benefits Sling RepoInit as a provisioning method
• Compare Sling RepoInit to other methods of initializing a repository
• Show how to manage permissions, configurations, and initial content using RepoInit
• Demonstrate using Sling RepoInit for both AEM as a Cloud Service and AEM 6
• Discuss a method for making it easier to manage Sling RepoInit configurations and future options 

More info about Repoinit: https://sling.apache.org/documentation/bundles/repository-initialization.html 

This talk will be useful for AEM Technical Experts, Architects, and Developers, especially those interested in AEM as a Cloud Service.

Register Here