I've only used Ant in a few projects, but it seriously sucks. It's hard to debug, has crappy error messages, is awkward to extend, has ugly, inflexible XML syntax. I find it much easier and faster to write build scripts in Java or Scala, so I would definitely take a look at your build tool.
Brian Clapper
· 10 months ago
Jesper, I've used Ant quite a bit. I've even written Ant tasks on occasion. I'm with you 100%. Martin Fowler has it right: A build tool should give you easy access to the full power of a real programming language. The syntax also should be easier to deal with than Ant's convoluted XML-based mess.
Glen Coakley
· 10 months ago
I am definitely interested in such a tool. We currently have python embedded in our scripts (Though, I doubt most developers here know it.) and there are many tasks that I would like to have but, that, well, would required Ant Tasks.
I think you are correct on the build language requirements and Ant only does #1 reasonably. At #2 it sucks like a black hole.
Off topic. I started to ask you, "What tools you use to create your blog and discussions?" but, (I'm a geek.), I just figured it out.
Erik Tjernlund
· 10 months ago
Check out Schmant and you can use Scala (or any other jvm available scripting language): Schmant It does need a task for building scala code though ...
Brian Clapper
· 10 months ago
Erik,
Schmant definitely looks interesting. However, Scala isn't (strictly speaking) a scripting language; it's a compiled language. I may find uses for Schmant, but I'm still going to implement this tool, even if it never gets used by anyone else, because it's an excellent exercise for bringing myself further into the Scala world.
Thanks for the tip, though. I've bookmarked Schmant.
Erik Tjernlund
· 10 months ago
I know Scala is compiled to byte code. You can use Schmant through compiled code as well. It's written in java and you can use the api via java (etc etc etc).
Brian Clapper
· 10 months ago
Got it. Thanks.
Sean Crotty
· 10 months ago
I'd love to see a few examples of Ant snippets that demonstrate particularly distasteful use compared with equivalent snippets using your Scala build tool (when it's more fleshed out).
Brian Clapper
· 10 months ago
Sean, as I get this thing built, I plan to blog about it, so you'll get your chance. ;-) In the meantime, though, compare an Ant build.xml file to a Rake build file; it'll give you a flavor for the type of approach I'm attempting to use.
Daniel Spiewak
· 10 months ago
Have you considered SBT? It's basically what you just wrote, except with some more time in grade. Personally, I use Buildr for my Scala projects. It's not a Scala-based build script specification (it's based on Rake), but it is quite concise and manageable.
Brian Clapper
· 10 months ago
SBT is very interesting indeed; I've bookmarked it. Thanks.
I think I'll continue to build the one I'm building, mostly because it's serving as a perfect platform for delving into various aspects of Scala--including Actors, which I'm using to parallelize the build process. When I'm teaching myself a new language, having a well-defined, real project to build is ideal and beats writing small test programs here and there. I'm also looking to build something I can use in my Java-only projects, so it should build more than just Scala projects.
But I believe I'll switch to SBT to build it, until it can build it with itself.
Build scripts should be written using real languages. I really like what buildr does http://buildr.apache.org/ The only drawback of this solution is a speed - for huge project it works very slow. If you make something similar but faster - that would be killer tool.
Jonhnny Weslley
· 10 months ago
Great idea!! Like as Rake and Maven your Scala build tool could manage dependencies and provide support to plugins!
Joerg
· 10 months ago
I'd be interested to hear, what you think about Gradle. Actually, I'd find a tool like Scadle much more appealing...
Brian Clapper
· 10 months ago
Given the number of build tools out there that are hosted on the JVM, evidently I'm not the only person who's disenchanted with Ant.
grh
· 5 months ago
In the groovy world, gant is to ant as gradle is to maven, except that where maven locks you pretty much in to the maven way, gradle provides sensible convention-over-configuration defaults but makes it trivial to change them.
It's a piece of work.
I can recommend you look at it, especially the sort of two-phase lifecycle it has, where first builds a di-graph of targets BEFORE any are executed. The script can generate or modify tasks and dependencies on the fly, and when it's all done, you get a chance to look at the graph before execution. Also it uses apache ivy natively for dependency mgmt, although it has a groovy dsl to configure it if you're XML averse (as I am).
All configured in pure groovy code.
Now a system that was similar, but configured in SCALA, and which in addition takes advantage of scala to be as parallel as possible, now THAT would be sweet.
Brian Clapper
· 5 months ago
Thanks. I'll give gradle a good look. My build tool has stalled, as other things have taken priority (and stolen my interest). I'm still using ant, which I find more objectionable by the day. Meanwhile, every time I start to dive into Maven, I think, "Seriously? Are you kidding me?" Something maven-like that uses a sensible configuration file format—especially one that makes it easy to escape to real code—would be a real win. Thanks for the pointer.
kramerica
· 5 months ago
dang nabit -- i was hoping to see some progress in build systems. in my opinion what the java community did was throw the baby out with the bath water when the ignored the essence of 'make'. what is really missing is a good approach to describe a build recipe. Unfortunately the java world is web centric when it comes to build system, so if you company is developing suites of products, unbundled products, products targeted at multiple channels: OEMs, VARs, ISV, IHV, Web, Direct these modern build tools are actually poor engineering projects. I just glanced at SBT and I'll need to muck with it. As soon as I see a build system that is not able to describe lists in a simple manner, I wonder what hoops the author will come up with to deal with variants. For example in 'make' I could do things like
LIST = foo LIST += bar
when I start to see folks implementing build systems like scons, and rake that use the native language syntax to express the recipe, I become concerned that they are really scripting the recipe and not defining the recipe
One of my favorite things about 'make' is the multiple pass thru the recipe to support interpolated macros, and exactly the opposite of ant/maven and that is mutable properties.
Every build system I've worked on where the developers started going outside the declarative and into procedural coding, the build system turned into a rats nest that nobody else on the team wanted to have anything to deal with.
having said all that, I'd love to see a build system implemented in scala, but designed/implemented with right reasons behind them. Often all I see from the author is a description of symptoms they are solving. What is needed is a paradigm change, not just more rehashing of the same old junk
Brian Clapper
· 5 months ago
Valid points. However, Martin Fowler's comments still rang true for me. Every build system I have ever used (and I used "make", in various forms, for many, many years) ultimately had to resort to code of some kind, because the standard, built-in, easy-to-read recipes didn't handle some obscure, project-specific thing or other. With "make", once you try to do anything halfway complicated, the Makefile rapidly becomes hard to maintain. I've written some pretty hairy GNU Make macros, solely to handle some odd, local build weirdness. I've also written some obnoxious-looking shell logic, for similar reasons.
Being able to escape to real code, when necessary, is helpful. Sure, someone will produce hairy crap because they have access to that much power. But they'll produce hairy crap with "make" or Ant, too. If the situation calls for actual code, I'd like to have access to an actual programming language.
kramerica
· 5 months ago
Great statements. But those are the symptoms that Martin talks about, not root cause. In fact many of the continuous integration things Martin writes about... are too late in the game for many commercial software companies. What Martin proposes in his writings is to create a symptom identification process via continuous integration, when in fact what you want to do is create a scenario where integration rarely fails. To do this, requires the right source code control tools, engineering discipline in the form of pre-integration practices, and a well designed build and test system. We were doing things like this back in the mid-80's on the HP-UX OS teams (specifically IND). At Sun in the early to mid-90's on some teams we ran a similar model. I've been lucky enough to work with distributed source code control tools in one form or another since the about 1987. They allow you to have workflows that are much simpler to use than the old-school cvs, subversion, and perforce. I've been creating java build systems since 1995, initially using make, then Ant, and more recently Maven. I thought nothing could get worse than Ant until I started working with Maven (i started drinking heavily). Scons I thought was going in the right direction, but really is no different than using groovy. I've read about a number of other approach, but they too fall into the same trap of too much coding in the build system. What is needed is a good external DSL with different thinking about the fundamental problems than in previous build systems
But build systems are generally a mess because of a lack of good examples available to learn from and people not dealing with the fact of a bad data model requires oddities to occur to operate on that data model
1) bad data model. generally speaking native source trees in commercial companies are poorly designed and rarely take into account the entire product life cycle. this leads to the problems Martin and others run into needing all kinds of 1-off solutions to produce a build. For example, suppose nothing changes in the source code, but the dependencies change, can the build system handle that without branching or modifying the build system
2) Ant and Maven are weakness is due to an very very narrow usage focus. Because of this focus and because they are the status quo, people are not learning much by using them, only that they do not do what is needed (and mostly what is needed is a reaction to a symptom). So in many ways because our build tools are weak so is our understanding of build systems.
3) Many java projects cannot fit in the maven model and the ant stuff is way to brutal in xml and also its really a poor design to begin with. I worked on one team that developing software to run on set top boxes (interestingly enough... javas initial target) and it was so painful using Ant to implement the build system. There is one good thing I can say about Maven and Ivy... the dependency management is good.
In regards to your latest statement about having to use actual code, I agree 100%. How that is done is the difference between what generally ends up being very sloppy build system or elegant (which should translate into easy to maintain and extend). I like to work with teams to completely separate targets, properties, and code which allows users and maintainers to focus on the specific logic of the build system versus, configuration of the build system, or supporting tasks (code snippers), instead of unraveling
In summary, when I define/implement build system, my main focus is removing the odd, local build weirdness. Mostly this oddness occurs for these reasons:
* the build system was difficult to understand and the developer needed to deliver their technology (primary focus) and in a flash put in something to keep progress moving forward
* the build system is much like a wiki and that is it often not owned by anyone, no specs, no best-practices, its just a unmanaged container that somehow produces a result
* the person assigned to developing the build system operated in much the same way agile teams do, and that is with a narrow focus on the goals of the team in the immediate and near future, and then iteratively add support. but generally fundamental flaws are established with initial instance and we pile on more tasks for the build system in rapid fire.
* garbage in, process garbage, maybe something useful as output.
The best thing we could do, is as a community establish a place on the web to describe the problems, symptoms, exceptions and approach to this problem. The solution would not be a build system, but instead of collection if libraries, utils (a toolbox), from that build systems would be implemented.
How simple can a native build system be... All targets are dynamic as follows and maintain in its own file lets call that targets.mk. Then we'll need another file to contain all the properties, properties.mk. The approach with that file is to define 3 categories of properties: little tuple, big tuple and common properties. The tuple type properties are used to group properties along platform, toolchain and other variants. The next file we'll next is responsible from resloving the properties into the variables used in the tarket.mk file. What we've done is separate the build system into 3 major pieces (targets, properties, resolver). This approach requires very few 'make' statements/commands. It support unlimited toolchains, chipsets, languages, etc... This is what I call a variant/relocatable based build system. Almost impossible to do in a simple manner with Ant and Maven.
$(OBJDIR)%.c: (SRCDIR)%.o $(DOT_C_DOT_O_RULE)
$(LIBRARY): $(OBJS) $(LIBRARY_RULE)
$(EXECUTABLE): $(OBJS) $(EXECUTABLE_RULE)
As soon as I see a target like libfoo.a I no that build system is rigid and will not respond to have the company does business, where they do it, etc.. without someone hacking on it.
So... A build system implementers, why don't we remove the odd and build weirdness? And why don't we work with build technology designers to have a better perspective of the problem?
Regards -- Kramer
kramerica
· 5 months ago
I've been thinking more about this topic over the past few days, and I wanted to toss out this thought and see what other folks thought. Ivy is a good example of a focused component that can be used in conjunction with other tools (would like to have a better interface other than XML). What if all the components in the build system were actually independent components. Not plugins, but actual independent components. If this approach was used for build systems, continuous integration systems, issue tracking systems, and wiki, I think we'd wind up with a collection of power single focused components that can be combined in many different ways to solve a far wider range of problems and people would be able to assemble solutions that are much more effective than lock in to a specific language and description model. In a proof-of-concept project I was working on during 2000-2002 (off and on), I was applying the same concept but with web apps, and that each application is independent but I could assemble a custom system per the organizations requirements: http://qbal.mozdev.org/oldQbal.jpeg and http://qbal.mozdev.org/ (I shut this effort down as other business activities got in the way). I'm going to restart it again, but this time, I'm going to start with these 2 areas. First will be a release definition tool that is generalized and its role is to provide a mechanism to define a release, its components, branches, automation properties, build properties, etc... from this automation is configuration, initialized and started-up. The next problem area I want to start on is the build system problem. What I'm looking to do is integration, and developer where needed a very loosely couple tool box of build components, i.e Ivy is a very good starting point, although I'd love for it to have an external DSL other than XML as the interface.
adsl viettel
· 4 months ago
oh, that's a geat idea, thanks you for your helpful post, i will follow your rss now.
andrew cooke
· 3 months ago
read this post while trying to understand sbt. what surprised me was the statement that you cannot override operators in python. you can - there are double-underscore methods for all of them. i have a dsl (for a parser - called lepl) that does this, implemented via a mixin. each object in your dsl simply pulls in that mixin.
All Comments
I've only used Ant in a few projects, but it seriously sucks. It's hard to debug, has crappy error messages, is awkward to extend, has ugly, inflexible XML syntax. I find it much easier and faster to write build scripts in Java or Scala, so I would definitely take a look at your build tool.
I think you are correct on the build language requirements and Ant only does #1 reasonably. At #2 it sucks like a black hole.
Off topic. I started to ask you, "What tools you use to create your blog and discussions?" but, (I'm a geek.), I just figured it out.
Schmant definitely looks interesting. However, Scala isn't (strictly speaking) a scripting language; it's a compiled language. I may find uses for Schmant, but I'm still going to implement this tool, even if it never gets used by anyone else, because it's an excellent exercise for bringing myself further into the Scala world.
Thanks for the tip, though. I've bookmarked Schmant.
I think I'll continue to build the one I'm building, mostly because it's serving as a perfect platform for delving into various aspects of Scala--including Actors, which I'm using to parallelize the build process. When I'm teaching myself a new language, having a well-defined, real project to build is ideal and beats writing small test programs here and there. I'm also looking to build something I can use in my Java-only projects, so it should build more than just Scala projects.
But I believe I'll switch to SBT to build it, until it can build it with itself.
Thanks to both of you.
Build scripts should be written using real languages. I really like what buildr does http://buildr.apache.org/ The only drawback of this solution is a speed - for huge project it works very slow. If you make something similar but faster - that would be killer tool.
It's a piece of work.
I can recommend you look at it, especially the sort of two-phase lifecycle it has, where first builds a di-graph of targets BEFORE any are executed. The script can generate or modify tasks and dependencies on the fly, and when it's all done, you get a chance to look at the graph before execution. Also it uses apache ivy natively for dependency mgmt, although it has a groovy dsl to configure it if you're XML averse (as I am).
All configured in pure groovy code.
Now a system that was similar, but configured in SCALA, and which in addition takes advantage of scala to be as parallel as possible, now THAT would be sweet.
LIST = foo
LIST += bar
when I start to see folks implementing build systems like scons, and rake that use the native language syntax to express the recipe, I become concerned that they are really scripting the recipe and not defining the recipe
One of my favorite things about 'make' is the multiple pass thru the recipe to support interpolated macros, and exactly the opposite of ant/maven and that is mutable properties.
Every build system I've worked on where the developers started going outside the declarative and into procedural coding, the build system turned into a rats nest that nobody else on the team wanted to have anything to deal with.
having said all that, I'd love to see a build system implemented in scala, but designed/implemented with right reasons behind them. Often all I see from the author is a description of symptoms they are solving. What is needed is a paradigm change, not just more rehashing of the same old junk
Being able to escape to real code, when necessary, is helpful. Sure, someone will produce hairy crap because they have access to that much power. But they'll produce hairy crap with "make" or Ant, too. If the situation calls for actual code, I'd like to have access to an actual programming language.
But build systems are generally a mess because of a lack of good examples available to learn from and people not dealing with the fact of a bad data model requires oddities to occur to operate on that data model
1) bad data model. generally speaking native source trees in commercial companies are poorly designed and rarely take into account the entire product life cycle. this leads to the problems Martin and others run into needing all kinds of 1-off solutions to produce a build. For example, suppose nothing changes in the source code, but the dependencies change, can the build system handle that without branching or modifying the build system
2) Ant and Maven are weakness is due to an very very narrow usage focus. Because of this focus and because they are the status quo, people are not learning much by using them, only that they do not do what is needed (and mostly what is needed is a reaction to a symptom). So in many ways because our build tools are weak so is our understanding of build systems.
3) Many java projects cannot fit in the maven model and the ant stuff is way to brutal in xml and also its really a poor design to begin with. I worked on one team that developing software to run on set top boxes (interestingly enough... javas initial target) and it was so painful using Ant to implement the build system. There is one good thing I can say about Maven and Ivy... the dependency management is good.
In regards to your latest statement about having to use actual code, I agree 100%. How that is done is the difference between what generally ends up being very sloppy build system or elegant (which should translate into easy to maintain and extend). I like to work with teams to completely separate targets, properties, and code which allows users and maintainers to focus on the specific logic of the build system versus, configuration of the build system, or supporting tasks (code snippers), instead of unraveling
In summary, when I define/implement build system, my main focus is removing the odd, local build weirdness. Mostly this oddness occurs for these reasons:
* the build system was difficult to understand and the developer needed to deliver their technology (primary focus) and in a flash put in something to keep progress moving forward
* the build system is much like a wiki and that is it often not owned by anyone, no specs, no best-practices, its just a unmanaged container that somehow produces a result
* the person assigned to developing the build system operated in much the same way agile teams do, and that is with a narrow focus on the goals of the team in the immediate and near future, and then iteratively add support. but generally fundamental flaws are established with initial instance and we pile on more tasks for the build system in rapid fire.
* garbage in, process garbage, maybe something useful as output.
The best thing we could do, is as a community establish a place on the web to describe the problems, symptoms, exceptions and approach to this problem. The solution would not be a build system, but instead of collection if libraries, utils (a toolbox), from that build systems would be implemented.
How simple can a native build system be... All targets are dynamic as follows and maintain in its own file lets call that targets.mk. Then we'll need another file to contain all the properties, properties.mk. The approach with that file is to define 3 categories of properties: little tuple, big tuple and common properties. The tuple type properties are used to group properties along platform, toolchain and other variants. The next file we'll next is responsible from resloving the properties into the variables used in the tarket.mk file. What we've done is separate the build system into 3 major pieces (targets, properties, resolver). This approach requires very few 'make' statements/commands. It support unlimited toolchains, chipsets, languages, etc... This is what I call a variant/relocatable based build system. Almost impossible to do in a simple manner with Ant and Maven.
$(OBJDIR)%.c: (SRCDIR)%.o
$(DOT_C_DOT_O_RULE)
$(LIBRARY): $(OBJS)
$(LIBRARY_RULE)
$(EXECUTABLE): $(OBJS)
$(EXECUTABLE_RULE)
As soon as I see a target like libfoo.a I no that build system is rigid and will not respond to have the company does business, where they do it, etc.. without someone hacking on it.
So... A build system implementers, why don't we remove the odd and build weirdness? And why don't we work with build technology designers to have a better perspective of the problem?
Regards -- Kramer