Introduction
The utPLSQL extension for SQL Developer was originally written in Xtend. In this blog post I explain why we decided to migrate the project to Java and how we’ve done that.
Why Replace Xtend?
Xtend is a statically typed, clean language with excellent string templating features and a fine integration into the Eclipse IDE. From a technical point of view it is an excellent choice for code generation projects, especially if you are a happy user of the Eclipse IDE.
In the Xtext release notes for version 2.20.0 – released in December 2019 – you will find the following statement about “Xtend”:
A word on Xtend. Back in 2013 Xtend was the “Java 10 of today” even before Java 8 was out. Meanwhile Java Release cadence has speeded up and many of Xtends features can be achieved with pure Java, too. There is still some areas where Xtend is particularly advanced like Code generation, Unit tests and lambda heavy APIs like JvmModelInferrer and Formatter. For other tasks there is no need to use Xtend. Also the resources we have no longer allow us to keep the head start against Java. And learning Xtend still is a burden for new Xtext users. To reflect this changed situation we have decided to make Java the default in the wizard again (except e.g. the Generator and a few other APIs). You can still decide if you want Java or Xtend in the workflow.
The situation of Xtend has not improved. Quite the contrary. In the release notes of Xtend 2.22.0 – released in Juni 2020 – you find the following statement prominent at the beginning:
As you might have recognized, the number of people contributing to Xtext & Xtend on a regular basis has declined over the past years and so has the number of contributions. At the same time the amount of work for basic maintenance has stayed the same or even increased with the new release cadence of Java and the Eclipse simultaneous release. Briefly: The future maintenance of Xtext & especially Xtend is at risk. If you care, please join the discussion in https://github.com/eclipse/xtext/issues/1721.
The above mentioned GitHub issue was created in late March 2020 and referenced on Twitter. My current assessment is that Xtext will survive and Xtend will eventually die. This assessment was the main driver to think about replacing Xtend in the utPLSQL code base.
Why Java?
SQL Developer runs on a JVM (current LTS versions 8 and 11 are supported only). Hence, extensions have to be written in a JVM language. The utPLSQL extension generates some code and SQL statements. Therefore a language supporting multiline strings would be helpful. Options are
- Kotlin
- Scala
- Groovy
- Clojure
- Java (version 15 introduced the final version of Text blocks, the first LTS version with this feature will probably be 17, expected in September 2021)
However, the support for multiline strings in all these languages is inferior to what Xtend provides. The template expressions of Xtend are extremely powerful. It’s a statically typed code templating language after all (sigh, there is no adequate replacement for that).
I had a closer look at Kotlin, since the grammar has a lot of similarities with Xtend and it seems to be the rising star according to the JVM ecosystem report. However, it requires a runtime library and the support of the language is very much IDE dependent. As with Xtend, just the other way around. This means, excellent support in IntelliJ IDEA, but significantly weaker in the Eclipse IDE.
At this point I stopped evaluating other JVM languages and decided to go with Java for the utPLSQL extension. The support of Java is excellent in any Java IDE. No wonder, Java is by far the most popular JVM language. Furthermore Java is used in the other utPLSQL projects (Java API, CLI and Maven plugin). This simplifies the contribution to the project. And last but not least I expected the least effort for the migration when using Java as the target language.
Migration Approach
Every Xtend file in the utPLSQL project is migrated. The following Nassi-Shneiderman diagram visualizes the migration approach.
The blue box “More Xtend Files?” represents the loop over all .xtend files. All green actions are trivial. Only the orange action “Refactor Java Source” is laborious.
At the very end you can remove Xtend and all its dependencies from your project build file (e.g. the Maven pom.xml).
I will explain every action for a Xtend source file in the next chapters. I used the Eclipse IDE for the migration process.
Copy Generated Java Source
Xtend compiles to Java 8 source code. Select Open Generated File from the context menu of the Xtend editor. Copy the complete generated Java code to the clipboard.
Rename .xtend to .java
Select the .xtend file in the Package Explorer, select Refactor -> Rename from the context menu and change the extension to .java. This will delete the generated Java file. That’s why we saved it to the clipboard before.
Paste Generated Java Source
Open the .java file in the editor (still containing the Xtend source) and replace the content with the one in your clipboard.
Now it’s a good time to save the changes in the version control system. Git in my case. I committed the rename and the content change. This way I still have access to the full history of the file. To the old Xtend version(s) and the Java source code generated by Xtend.
Refactor Java Source
Technically, we’re done. Unfortunately the generated Java code rarely looks as if you wrote it by hand. We have to refactor the code to make it maintainable. Furthermore we want to eliminate all dependencies to the Xtend runtime library. In this project, we went a step further and eliminated all dependencies to Eclipse libraries including their dependencies.
Open the Java file and the original Xtend file (based on the Git history) side-by-side. Then apply the actions outlined in the next chapters.
Step 1 – Fix Copyright Comment
In this project we define a copyright header in each file as a normal comment. This means with /* ... */ . The generated Java file converted them to Javadoc style comments (/** ... */). Remove the superfluous *.
Step 2 – Remove Xtend Annotations
In our case the generated Java file contained the following Xtend specific annotations:
@SuppressWarnings("all")@Extension@Accessors@Pure
Remove these annotations. They are not needed. However, there will be a lot of warnings. This is good. We have to address them eventually.
Step 3 – Format
Format the whole Java file with your favorite formatter settings.
Step 4 – Remove Class References
The generated Java code references static methods and fields with the class name. That’s fine for other classes. But I do not like it for the own class. Here an Example:
progressBar.foreground = GREEN
this.progressBar.setForeground(RunnerPanel.GREEN);
progressBar.setForeground(GREEN);
Search for the Classname followed by a dot (.) and replace it with nothing. This may lead to compile errors. E.g. when initializing the logger field. Add the class reference again to fix these kind of errors. Of course, you could also decide for each occurrence whether a replacement is useful. I tried that as well, but found that it was easier to fix the few errors afterwards.
Step 5 – Remove this. References
The generated code references all fields and instance methods with this. I do not like that either. Therefore I replaced all occurrences of this. with nothing. Then I fixed the compile errors by I adding the this. where it was really necessary. I found it easier and less error-prone to do it this way.
Step 6 – Fix Fields
The generated Java file has blank lines between fields. Remove the unwanted blank lines. Use the original Xtend file as reference. This’s why we opened the original Xtend file before, side by side with the Java file.
Step 7 – Fix Constructors, Methods
Fix each constructor and method individually. For me, a method was a good unit of work. Follow the steps in the next subchapters.
Step 7.1 – Add Missing Comments
Javadoc comments are part of the generated Java file. But all other comments are missing. You have to copy these comments from the original Xtend file to the Java file. The side-by-side view is really helpful for this step.
Step 7.2 – Eliminate Variables Beginning with an Underscore (_)
All variables starting with an underline are intermediate results of an expression. Here’s an Example:
ret = formatter.format(seconds / 60 / 60) + " h"
String _format = formatter.format((((this.seconds).doubleValue() / 60) / 60)); String _plus = (_format + " h"); ret = _plus;
ret = formatter.format(seconds / 60 / 60) + " h";
In this case, you could simply copy the original Xtend expression to Java. However, this is not always possible and I found it error-prone. For me it was much safer to edit each variable individually. In this case this means:
- Copy the expression of the variable
_formatto the clipboard - Replace the usage of
_formatin the expression of the variable_pluswith the clipboard content - Copy the expression of the variable
_plusto the clipboard - Replace the usage of
_plusin the expression of the variableretwith the clipboard content - Remove the now unused variables
_formatand_plus - Simplify the expression, remove unnecessary parts (see also next chapters)
Step 7.3 – Eliminate Unnecessary Parentheses
Boolean parts of an expression are surrounded by additional parentheses in the generated Java file. Remove these superfluous parentheses. Here’s an example:
if (desktop !== null
&& desktop.isSupported(Desktop.Action.BROWSE)
&& url !== null)if ((((desktop != null)
&& desktop.isSupported(Desktop.Action.BROWSE))
&& (url != null)))if (desktop != null
&& desktop.isSupported(Desktop.Action.BROWSE)
&& url != null)Step 7.4 – Eliminate Unnecessary else Branches
else if branches are converted to else { ... if constructs in the generated Java file, if the Xtend generator creates an intermediate variable. Simplify these constructs. Here’s an example:
if (itemType == "pre-run") {
event = doc.convertToPreRunEvent
} else if (itemType == "post-run") {
event = doc.convertToPostRunEvent
} else if (itemType == "pre-suite") {
event = doc.convertToPreSuiteEvent
} else if (itemType == "post-suite") {
event = doc.convertToPostSuiteEvent
} else if (itemType == "pre-test") {
event = doc.convertToPreTestEvent
} else if (itemType == "post-test") {
event = doc.convertToPostTestEvent
}boolean _equals = Objects.equal(itemType, "pre-run");
if (_equals) {
event = this.convertToPreRunEvent(doc);
} else {
boolean _equals_1 = Objects.equal(itemType, "post-run");
if (_equals_1) {
event = this.convertToPostRunEvent(doc);
} else {
boolean _equals_2 = Objects.equal(itemType, "pre-suite");
if (_equals_2) {
event = this.convertToPreSuiteEvent(doc);
} else {
boolean _equals_3 = Objects.equal(itemType, "post-suite");
if (_equals_3) {
event = this.convertToPostSuiteEvent(doc);
} else {
boolean _equals_4 = Objects.equal(itemType, "pre-test");
if (_equals_4) {
event = this.convertToPreTestEvent(doc);
} else {
boolean _equals_5 = Objects.equal(itemType, "post-test");
if (_equals_5) {
event = this.convertToPostTestEvent(doc);
}
}
}
}
}
}if ("pre-run".equals(itemType)) {
event = convertToPreRunEvent(doc);
} else if ("post-run".equals(itemType)) {
event = convertToPostRunEvent(doc);
} else if ("pre-suite".equals(itemType)) {
event = convertToPreSuiteEvent(doc);
} else if ("post-suite".equals(itemType)) {
event = convertToPostSuiteEvent(doc);
} else if ("pre-test".equals(itemType)) {
event = convertToPreTestEvent(doc);
} else if ("post-test".equals(itemType)) {
event = convertToPostTestEvent(doc);
}Step 7.5 – Eliminate Unnecessary Objects.equal Usages
Xtend supports equality operators (==) and implements that with Google’s Objects.equal in the generated Java file. Since we want to eliminate all Xtend dependencies and we are not planning to use Guava as an additional dependency, we replace the comparison with a plain Java construct. See the previous example in step 7.4.
Step 7.6 – Eliminate Unnecessary Code Blocks
The generated Java file may contain unnecessary code blocks {...}. Remove them. Here’s an example.
val failedExpectations = node.getNodeList("failedExpectations/expectation")
for (i : 0 ..< failedExpectations.length) {
val expectationNode = failedExpectations.item(i)
val expectation = new Expectation
event.failedExpectations.add(expectation)
expectation.populate(expectationNode)
}final NodeList failedExpectations = this.xmlTools.getNodeList(node, "failedExpectations/expectation");
int _length = failedExpectations.getLength();
ExclusiveRange _doubleDotLessThan = new ExclusiveRange(0, _length, true);
for (final Integer i : _doubleDotLessThan) {
{
final Node expectationNode = failedExpectations.item((i).intValue());
final Expectation expectation = new Expectation();
event.getFailedExpectations().add(expectation);
this.populate(expectation, expectationNode);
}
}final NodeList failedExpectations = xmlTools.getNodeList(node, "failedExpectations/expectation");
for (int i = 0; i < failedExpectations.getLength(); i++) {
final Node expectationNode = failedExpectations.item(i);
final Expectation expectation = new Expectation();
event.getFailedExpectations().add(expectation);
populate(expectation, expectationNode);
}Step 7.7 – Eliminate StringConcatenation Usages
Xtend implements multiline strings with the help of the StringConcatenation class in the generated Java file. To eliminated Xtend dependencies we use either the Java + operator or the StringBuilder class. Here’s an example:
val sql = '''
SELECT table_owner
FROM «IF dbaViewAccessible»dba«ELSE»all«ENDIF»_synonyms
WHERE owner = 'PUBLIC'
AND synonym_name = '«UTPLSQL_PACKAGE_NAME»'
AND table_name = '«UTPLSQL_PACKAGE_NAME»'
'''StringConcatenation _builder = new StringConcatenation();
_builder.append("SELECT table_owner");
_builder.newLine();
_builder.append(" ");
_builder.append("FROM ");
{
boolean _isDbaViewAccessible = this.isDbaViewAccessible();
if (_isDbaViewAccessible) {
_builder.append("dba");
} else {
_builder.append("all");
}
}
_builder.append("_synonyms");
_builder.newLineIfNotEmpty();
_builder.append(" ");
_builder.append("WHERE owner = \'PUBLIC\'");
_builder.newLine();
_builder.append(" ");
_builder.append("AND synonym_name = \'");
_builder.append(UtplsqlDao.UTPLSQL_PACKAGE_NAME, " ");
_builder.append("\'");
_builder.newLineIfNotEmpty();
_builder.append(" ");
_builder.append("AND table_name = \'");
_builder.append(UtplsqlDao.UTPLSQL_PACKAGE_NAME, " ");
_builder.append("\'");
_builder.newLineIfNotEmpty();
final String sql = _builder.toString();final StringBuilder sb = new StringBuilder();
sb.append("SELECT table_owner\n");
sb.append(" FROM ");
sb.append(getDbaView("synonyms\n"));
sb.append(" WHERE owner = 'PUBLIC'\n");
sb.append(" AND synonym_name = '");
sb.append(UtplsqlDao.UTPLSQL_PACKAGE_NAME);
sb.append("'\n");
sb.append(" AND table_name = '");
sb.append(UtplsqlDao.UTPLSQL_PACKAGE_NAME);
sb.append("'");
final String sql = sb.toString();Step 7.8 – Eliminate Escaped Apostrophe (\') Usages
The generated Java file escapes every apostrophe. This is necessary for chars ('\''), but not for strings ("'"). We replace all escaped apostrophes (\') in strings with a plain apostrophe ('). This improves the readability, especially for SQL statements. See the example in step 7.7.
Step 7.9 – Eliminate Conversions Usages
Xtend implements collection literals (#[...] and #{...}) with the help of the Conversions class in the generated Java file. We replace all usages of this class with plain Java constructs. Here’s an example:
lov.put(RESET_PACKAGE, #[YES, NO])
lov.put(RunGenerator.RESET_PACKAGE,
Collections.<String>unmodifiableList(
CollectionLiterals.<String>newArrayList(
RunGenerator.YES, RunGenerator.NO
)
)
);lov.put(RESET_PACKAGE, Arrays.asList(YES, NO));
Step 7.10 – Eliminate all Exceptions Usages
Xtend does not force you to handle checked exception, instead it catches them in the generated Java file and throws an own RuntimeException using the Exceptions class. We replace all usages of this class with plain Java constructs. Here’s an example:
def getNodeList(Node doc, String xpathString) {
val expr = xpath.compile(xpathString);
val NodeList nodeList = expr.evaluate(doc, XPathConstants.NODESET) as NodeList
return nodeList
}public NodeList getNodeList(final Node doc, final String xpathString) {
try {
final XPathExpression expr = this.xpath.compile(xpathString);
Object _evaluate = expr.evaluate(doc, XPathConstants.NODESET);
final NodeList nodeList = ((NodeList) _evaluate);
return nodeList;
} catch (Throwable _e) {
throw Exceptions.sneakyThrow(_e);
}
}public NodeList getNodeList(final Node doc, final String xpathString) {
try {
final XPathExpression expr = xpath.compile(xpathString);
return ((NodeList) expr.evaluate(doc, XPathConstants.NODESET));
} catch (XPathExpressionException e) {
final String msg = "XPathExpressionException for " + xpathString + ".";
logger.severe(() -> msg);
throw new GenericRuntimeException(msg, e);
}
}Step 7.11 – Fix Return Type void
In Xtend everything is an expression. The return type of a method is inferred if it is not explicitly defined. Therefore the generated Java file might have determined a “wrong” return type when you do not really need one. Change the return type to void in such cases. Here’s an example:
def setSeconds(Double seconds) {
this.seconds = seconds
}public Double setSeconds(final Double seconds) {
return this.seconds = seconds;
}public void setSeconds(final Double seconds) {
this.seconds = seconds;
}Step 8 – Eliminate Remaining Usages of Classes Provided by Xtend
Check the import section of the Java file. Look for the following packages:
com.google.common.base.*org.eclipse.xtend2.lib.*org.eclipse.xtext.xbase.lib.*
Remove these imports and find a plain Java solution.
The replacement of ToStringBuilder was challenging. This class builds a nice String representation of all fields using Java Reflection. I thought about using Apache Commons or Project Lombok for my model classes. However, for the utPLSQL project I decided to use a solution without Java Reflection and without adding addition dependencies to the project. We already use the Spring Framework, mainly for JDBC. Therefore I used Spring’s ToStringCreator with a custom styler to represent the String as nicely formatted JSON.
Step 9 – Test
After these refactoring steps, it’s about time to run tests. I was glad to have unit tests with a reasonable coverage. Since this is an extension for SQL Developer, some things only work in the SQL Developer environment (e.g. the controller for actions in the context menu of the navigator etc.). By reasonable coverage, I mean for the code that can be executed outside of SQL Developer via JUnit tests.
So at this point I ran the unit tests and depending on the class I built the extension, installed it in SQL Developer and tried if the changed class behaved as expected. Of course, I found a couple places where I could or better should change things to improve the testability of the code. Maybe one day. But without unit testing I would never have found the confidence to do all these refactoring.
Summary
Using this approach I was able to migrate the complete utPLSQL for SQL Developer project. The migration was time-consuming, but not complicated. I think it was the right decision for the utPLSQL project. We have reduced the number of languages and therefore the overall complexity of the project. The code base is now independent of an IDE. I used IntelliJ IDEA to implement additional features for version 1.2.0 of the utPLSQL extension. And it worked like a charm. I’m sure that this will simplify the contribution to this project in the future.
But what does this mean for other projects that are using Xtend? Should they also migrate to Java? That really depends on the kind of project. In code generation projects, the code template classes written in Xtend are superior to other technologies. You really have to weigh the pros and cons. However, the fact that the future of Xtend is currently uncertain should not affect the decision. I have proven that it is possible to migrate from Xtend to Java in a reasonable time. Hence there is no reason to panic.
The post Bye bye Xtend, Welcome Java appeared first on Philipp Salvisberg's Blog.