The Common Opensource Application Publishing Platform (CoApp)

Listen up folks, this stuff is big.

Today, I’m announcing the beginning of a project that intends to bring a little joy into the hearts of Open Source aficionados on the Windows Platform.

The biggest challenge to using/building/maintaining many Open Source applications on Windows, is that Windows does a lot of things differently than Linux and Unix . Different filesystems, command lines, APIs, user experiences … well, pretty much everything. Regardless of personal opinions about it being the ‘right-way’ or ‘wrong-way’, it suffices to say that it is just simply different. 

In order to build an Open Source application like PHP for Windows from scratch, I need to have a collection of libraries created from a fair number of different projects.  This creates a dependency between the code that I’m working on—PHP—and the project that supplies the library that I need.  It’s pretty important that I not simply rely upon a previously compiled version of the library (provided either by the project itself, or a third party) for a number of reasons:

• I want to make sure that the library is compiled with the same version of the compiler and libraries as I use.
• In order to fine-tune performance, I’m going to need to change the compiler settings.
• As a security precaution against malicious third parties creating flawed binaries.
• Hey!--It’s Open source. It’s pretty much a moral imperative that I compile the code for myself. Well, it is for me anyway.

Now, unfortunately, those dependencies don’t necessarily share the same development environments, practices, tools, operating systems, or even ideas as to how things should—from one’s own perspective—be done (because, as every developer knows, one’s own way is the ‘one true way’).

Interestingly, this problem really doesn’t happen on Linux (and other *NIX-like substances).  When someone builds that same application (PHP) on Unix, they do so knowing that the OS works a certain way (generally speaking), and along with the dark magic known as autoconf, you can put the source code on nearly any Unix-variant and just build it.

When that same application needs to be built on Windows, it takes some effort. Finding the dependencies (like OpenSSL or zlib), and getting them to compile (which is inconsistent from library-to-library on Windows) and then building the application itself—again, inconsistent—generates a binary that you can run. Nearly all of the time, if someone posts those binaries, they bundle up their copies of the shared libraries along with the application.  The trouble is, that there is no common versioning, or really, sharing of shared libraries on Windows. If your app and my app both use the same library, they could (and often do) ship with a different version of it.

And, there is the user side of the equation…

Of course. Consumers of open source software on Windows have been relegated to manually scouring the Internet for binaries, and they are often out-of-date, compiled against older compilers and libraries, and pretty hard to get working. Clearly there is a strong need for a package management system, along the same lines as apt, rpm, synaptic (and others) but built for the Windows platform, and compatible with Windows features.

Why not adapt the Unix-way on Windows?

There are two fundamental reasons: Primarily, because it’s just not done that way on Windows.  And since Windows doesn’t “look” like Unix, it’s not very easy to use the same scripts on Windows as Unix. Sure, there are Unix-like environments for Windows (Cygwin, Mingw and Microsoft’s own SUA), but they really isolate the developer from Windows itself. While they do try to create a very Unix-like environment, you end up building Unix-style apps on Windows, and pretty much forego the platform benefits that are available.

Secondly, open source software that was originally written for Windows won’t be using Linux-style tools anyway. Since I want to unify these two groups, I’m going to want a one-size-fits-all solution.

Really, the solution is to build it right—for Windows.

So, what exactly does “Building it Right” mean anyway?

That is, in a nutshell, the sixty-four kilobyte question.

For starters, this means using the tools, methodologies and technologies on Windows, as they were meant to be used, in order to take advantage of everything that Windows has to offer. I’m not interested in simply making a knock-off of the Unix-style way of doing things. Windows doesn’t store binaries in c:\usr\bin (/usr/bin) and libraries in c:\usr\lib (/usr/lib), so we’re not going to do things like that.

CoApp will:

• Provide a distributed, community driven package management system for open source applications on the Windows Platform
• Handle multiple versions of binaries using WinSxS (I know, even the mention of side-by-side components evokes fear, anger and the desire to go off-diet, but bear with me, I think we’ve got a solution), including multiple copies of the same version of the same library, compiled with different compilers.
• Support 64 bit and 32 bit systems, without hassle or collisions.
• Place binaries, libraries and header files in a logical and consistent location.
• Have tools and methods for handling dependencies.
• Create reliable installer packages (MSIs) for installing open source software.
• Facilitate sharing of components and allow multiple projects to easily both participate and consume them.
• Allow for upgrades and patching of both libraries and applications.
• Be Windows developer friendly. No forcing of building using ‘make’, but rather taking advantage of the nifty IDEs we already have.
• Also be Windows admin friendly. Even if it’s open source, you shouldn’t have to be a developer to put Open Source applications on Windows.
• Use advanced optimization techniques like Profile Guided Optimization to produce optimized binaries.
• Support future technologies as they come along.
• Aid in the adoption of Windows Error Reporting (WinQual) to assist in making software run better on Windows.
• End the eternal struggle between Green and Purple. Unless of course you’re a Drazi and are conducting elections.

Tall order? You bet. Still, I believe that it’s all achievable. I’ve spent the last several months working on some proof-of-concepts, fleshing out some ideas, and talking with some open source community members. Nothing is currently set in stone, and even the specifications are very fluid at this point.

I’ve started a project on Launchpad at https://Launchpad.net/coapp and the wiki at https://CoApp.org. I’m just starting the specifications and tools to make this happen, and I welcome everyone’s input and contributions.

Comments

• Anonymous
  March 31, 2010
  The comment has been removed

• Anonymous
  March 31, 2010
  Well, that's what I'm here to change. Don't get me wrong, it's a big project, but I'm more than up to the challenge. It doesn't hurt that I've already built proof-of-concepts on how to do pretty much everything. I've also recruited folks from Apache, PHP, Python and other communities to help out with this. I hope that you'll be pleasantly surprised a year from now.

• Anonymous
  March 31, 2010
  I hope for great usability (and IDE Integration).

• Anonymous
  March 31, 2010
  @gdmk -> Me Too. Actually, it's good to point out that IDE integration is extremely important to me. I'm both kinds of developers: the kind that works without an IDE and the kind that loves works with an IDE. One of my primary goals is that developers who only like to work in an IDE are kept as happy as possible. :D

• Anonymous
  April 01, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  This announcement would've been bigger had you waited until you had something functional. How many times has Microsoft announced something like this only to let it fizzle out? How's Interix going? SFU? Oh and the LAST community site you guys had for this specific purpose? Remember what it was called? I don't. I remember the grand announcement, though. Actions speak louder than blogs.

• Anonymous
  April 08, 2010
  >> This announcement would've been bigger had you waited until you had something functional. Yeah, well, I need to be working with open source communities in order to get this accomplished. Waiting until I was done would be a bigger splash, but it would not allow the OSS community the opportunity to shape design and development. I've already got 5 non-Microsoft committers signed up. >>Oh and the LAST community site you guys had for this specific purpose? Remember what it was called? I don't. I remember the grand announcement, though. I have no idea what you might be talking about. Regardless, this isn't Microsoft's project--it's mine. They just happen to pay me to work on it.  

• Anonymous
  April 08, 2010
  GarrettS you are my savior! As a Windows open source developer, this could be a dream come true! I'd love to be able to get involved too! I'm apparently on an exclamation point binge, but I'm just really excited!

• Anonymous
  April 08, 2010
  @Eric, If you want to get involved, the project site is at http://launchpad.net/coapp and the Wiki is at http://coapp.org ... the wiki is a static mirror today, it's hosting being moved right now. However, sign up on Launchpad, join the mailing list there, and when the Wiki is read-write, register on there too. I'd more than welcome all the help you can provide.

• Anonymous
  April 08, 2010
  Sounds like an excellent idea! In a way, Windows already has a feature along these lines "Control Panel/Programs/Turn Windows Features on or off". Could this be extended to "Turn OSS Features on or off"?

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  Microsoft's pursuit of and willingness to prosecute software patents is a risk to all developers, OSS and otherwise. Why would I do anything to support their platform?

• Anonymous
  April 08, 2010
  I like the sound of the package management but the concept of 'no forcing the use of make' in favour of 'nifty IDEs' sounds like pure fail. If those nifty IDE's actually worked properly with ANSI code and didn't fail so badly in so many ways I'd agree but for now most unix software will require a GCC toolchain to build without errors. Starting off with an openly hostile position towards the de-facto OSS toolchain isn't going to help this project one bit.

• Anonymous
  April 08, 2010
  I get where you're coming from, and it makes sense. Just curious: would universal availability of PowerShell change one's approach to OSS compilation and deployment, or is that mostly irrelevant? (Sorry, I'm not primarily a Windows person, so this kind of thing is unclear to me.)

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  @Todd M. No, powershell doesn't really solve anything in this regard.

• Anonymous
  April 08, 2010
  Well they know the open source way, when we know the windows way when can we use windows' apps on open source unix, when ?????

• Anonymous
  April 08, 2010
  I think that you have a lot of work maybe for 5 years or maybe more, Im going to try to help to this platform. Congrats,

• Anonymous
  April 08, 2010
  @Moxi It better not take me that long!

• Anonymous
  April 08, 2010
  Props on the awesome Idea and the Babylon 5 reference. Do you plan on incorporating some of the Free-as-beer offerings like Visual Studio Express and XNA Studio Express / XNA Framework into the project or repositories? It seems like this could be not only Apt-for-Windows, but a one-stop-shop for doing any type of hobbyist development on Windows.

• Anonymous
  April 08, 2010
  @captjc Re: B5 -- thank you!! You are only the third person to even notice that. I thought it was the crowning acheivement of the post! ;D And yes. Eventually, that's exactly where I want to be. Just don't tell anyone.

• Anonymous
  April 08, 2010
  I think you are on the right track here. This will be extremely beneficial even with moderate success. There must have been thousands of times during my career (99% Unix / Linux) that I have wished something along these lines existed. Looking forward to this. I strongly feel the urge to call this a bridge. Because in real life, there exists a multitude of OS platforms including Windows. Your project is essentially about interoperability. Finally, the means to achieve that.

• Anonymous
  April 08, 2010
  Before you go off totally re-inventing the wheel go look at what the Strawberry Perl folks have done with providing a toolchain. They don't MSI, but it may provide a staring point for you to cut some development time. Or not. Just check it out.

• Anonymous
  April 08, 2010
  @Matthew I've looked at Strawberry Perl, the trouble is, I have a lot of scenarios to include, and I really really do need to use MSI. That being said, it's not as much development work as it sounds :D ...

• Anonymous
  April 08, 2010
  Err, hello! Please don't forget other developers too. I'd love to hook into a MS sanctioned software update mechanism for my apps. apt-get is the biggest benefit of Linux and if you guys get it across the board... Well, I'll have less to complain about.

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  @GarrettS, @"Matthew Persico" Actually, Strawberry Perl has an MSI installer now.

• Anonymous
  April 08, 2010
  This looks like great, look forward to seeing this! regards, Malcolm.

• Anonymous
  April 08, 2010
  Just to put this in Steve Ballmer's terms. Isn't this just ripping of all the "innovation" from the  open-source world? All joking aside, glad someone finally decided to do this for the Windows platform.

• Anonymous
  April 08, 2010
  All sounds great; consistancy of build, dependancy gathering and deployment/installation is always good. However - how does CoApp sit with WIX - another MS sponsored open source(?) installer/build application. P.

• Anonymous
  April 08, 2010
  I am the contributor of the horn project which has attempted to solve this probem: http://www.hornget.net/packages/ http://github.com/dagda1/hornget http://github.com/dagda1/horn_src If you have any questions, please contct me: dagda1@scotalt.net I have experience, I can share in this difficult field.

• Anonymous
  April 08, 2010
  Personally I would prefer a pkgsrc for msys/mingw with integration for codelite and/or codeblocks. More important would be a cross-platform pkgsrc around ant for java (jpackage is not for me and awfully) outdated. Since linux takes care of (1), I believe (2) is more important. Anyway, good luck.

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  I wonder if the Open Build Service would be interesting for this - I know the devs would love to support building win packages as well, maybe cooperation is a good idea? You'll have infrastructure right away, a pretty powerful buildfarm, lots of apps to be build and name recognition too...

• Anonymous
  April 08, 2010
  oh let me add that obs is on buildservice.org :D

• Anonymous
  April 08, 2010
  That's great news! There are far too many open-source programs on my Windows system that include copies of the same libraries, esp. gtk or qt. From a developer perspective, having to manage several library dependencies by hand is very time-consuming, especially as some libraries won't compile with Visual Studio -- most GNU libs, in fact. Compiling binutils with VS was simply not possible, it would have taken weeks to convert the configure to something saner. Having a community effort for distributing vcproj files for common libraries would be a godsend, as well as pre-built ones. By the way, the msi format is a slight pain but using WiX was very helpful and a rather straightforward process. I'd recommend providing both vcproj and WiX and MSI files.

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 08, 2010
  Windows is a lost cause -- good luck wallowing in futility for the next year.  Get on the boat, most people have been staring down at Microsoft's corpse since before Gates stepped down. Do you think no matter how well this project turns out (I expect epic fail) you can convince any real admin or developer that this is the right way to go ?

• Anonymous
  April 08, 2010
  Sounds great, could help the world. In the meantime I've been working with Python, which seems to have licked the majority of the issues mentioned and has a huge library of platform independent apps and interfaces.

• Anonymous
  April 08, 2010
  Nice Project but as mentioned allready..probably not all has to be reinvented... Are you aware of http://wix.sourceforge.net/index.html ? You probably just need to write the IDE for those XML Files and trigger the compile.

• Anonymous
  April 08, 2010
  Have a look at CMake please. You can find it at www.cmake.org It can create unix-style MakeFiles, Windows style IDE project files and using cpack it can package everything into RPM, DEB, ... and NSIS windows installer programms. I'm sure they wouldn't mind getting code for a MSI package generator. This could help not only windows folks but also the projects themself. Because let's face it ... nobody knows how that autoconf stuff works anyway.

• Anonymous
  April 08, 2010
  The comment has been removed

• Anonymous
  April 09, 2010
  Good idea, please implement an update mechanism.

• Anonymous
  April 09, 2010
  @KS -- An update mechanism is part of the design.

• Anonymous
  April 09, 2010
  That certainly looks interesting, and would make Windows a much more viable platform for me in the longer term (although I'd really prefer it if Windows was more POSIX/UNIX-like). I've subscribed to the mailing list, but just wanted to say, boy, do I hate your choice of hosting/VCS. Launchpad is the worst of the big DVCS hosting trio (launchpad, github, bitbucket), and is also much worse than code.google.com. bzr is also the least popular of the three big DVCSes (and the slowest)...

• Anonymous
  April 09, 2010
  The comment has been removed

• Anonymous
  April 09, 2010
  The comment has been removed

• Anonymous
  April 09, 2010
  Good effort. Are you aware of the KitSetup package manager leveraged by the Windows Driver and Logo Kit teams?

• Anonymous
  April 09, 2010
  The comment has been removed

• Anonymous
  April 09, 2010
  WinSxS? so what about non admin/single user installs?

• Anonymous
  April 09, 2010
  Great to hear that MS is jumping in to solving this. I have also spent some time thinking about this problem a bit. I wrote something of a proposal over here: http://wiki.opensuse.org/openSUSE:Build_Service_Concept_Windows_Support Maybe some of that is useful to you.

• Anonymous
  April 09, 2010
  The comment has been removed

• Anonymous
  April 09, 2010
  Looking forward, this project contains usefull points that our project could profit from.

• Anonymous
  April 10, 2010
  This sounds very interesting.  I have very little experience with programming so I"m not qualified to help but I run quite a few open source apps on windows.  I've thought for a while that it would be real useful to have some kind of open source app/package manager like apt in windows.  Then I could install programs I use from one place like I do with my ubuntu machines.  I like the sound of this.

• Anonymous
  April 10, 2010
  +100 for the idea. It is big. Missing apt-get/yum/zypper is what makes Windows inferior to say Linux for OSS development. -100 for trying to reinvent the wheel (build system) I'm putting here a big PRO for CMake. Guys at Kitware did an amazing job and had invested 10 or so years into it. Instead, build  or distribution service should build on existing technologies. RPM and Debian did not come with new "make" tool, they did not reinvent autotools, they can play nice with both autotools and cmake or make for that reason. I'm saying that if CoApp would be trying to both create a build system and apt-get it does not have a chance to be ready in this decade and to be accepted by the OSS community which is looking for a single tools that works anywhere. If CoApp only concentrates on "apt-get" part while adding some missing pieces to CMake e.g to support MSI, than chances to be ready and get accepted are much better IMHO.

• Anonymous
  April 10, 2010
  @Vladislav -- I'm not building a new build system. I'm using Visual Studio (MSBuild) What I am building is tools to transform whatever existing build process to one that Visual Studio supports. I spoke to Bill Hoffman (Kitware) YESTERDAY about this very thing. Now, luckily, the tool that actually generates the .vcxproj/.sln files is completely pluggable, meaning that with very little effort, it could churn out CMake scripts. Bill and I are going to sync up again over the next few weeks to see where we can accomplish this.

• Anonymous
  April 10, 2010
  The comment has been removed

• Anonymous
  April 10, 2010
  Hi Garrett, This sounds like a great idea. Lets hope it works out! Got a couple of questions for you: Will you also make sure it is non-admin freindly? That is that software that is not say a webserver, but is more user space, will not hassle the user for admin rights unnecessarily with UAC? If it works with VC, can you also make sure it is VC express compatible for hobbyists? It's one thing having an employer fork out for VS at work, but another doing so at home, especially when GCC + Eclipse is free. One other thing worth noting is that the usual add/remove programs dialogue in Windows, while the de-facto way of removing non system stuff, is one big flat list, which when many libraries and tools with dependencies are installed will look ugly. Perhaps one idea is to have in the dialogue a single CoApp bundle, with a "Change" button that launches a wizard that groups up packages (as do most Linux package managers), and handles dependencies. Heck - this could be used to add as well as remove packages if linked with update sources in a similar way to portage/apt/yum. Danny

• Anonymous
  April 10, 2010
  Interesting article, but I think you have a number of problems to overcome, and it's not clear to me if you're trying to create a standardised compiling mechanism (like portage) or a way of standardising the installation of open-source apps (apt/yum). Either way, I feel that Windows isn't quite ready for it - with the way directory structures exist with Windows and the challenge of getting vendors onboard. Good luck with it, though. The concepts themselves are sound - I'd like to see this succeed.

• Anonymous
  April 11, 2010
  The comment has been removed

• Anonymous
  April 11, 2010
  Hmm - couldn't stop myself. Windows will not be ready until somebody like Garrett does this - I think full steam ahead. I understand the distinction you are making between binary package managers like Apt/yum and source/build packaging like portage, however, for this project it looks like it will be trying to bring both things together, which I for one think is great. Having been a Gentoo user, as well as Suse and Ubuntu (yes also Redhat and Debian), I am familiar with them, and always would have loved to combine some of the strengths of both types of systems.

• Anonymous
  April 12, 2010
  Yay, at last! I've been saying somebody should do this for years... Some of my perspective:

• Yay for using MSI - presumably you include some kind of extra information in MSI files about their dependencies?

• As somebody mentioned above, it would be great it you supported an opensource toolchain as well as the Microsoft one

• Even if you're using the Microsoft toolchain (and I understand the desire to have people using IDEs be happy), please make it so auto-building a source package is a single-step operation that doesn't have to involve opening the IDE - this should be doable with the Microsoft make tools. Essentially you want to be able to say "get me the dependencies of this package, run the build, give me the output" without lots of clicking.

• What's your strategy for dependencies? Although the target platform's totally different, apt-get and yum solve lots of finicky dependency problems - I think re-writing your own system for finding and deploying packages and resolving dependency issues is going to be more work than building on top of what's already there

• For people building commercial software on top of open source software, it would be great to be able to have additional repositories that can depend on libraries in the open ones

• Anonymous
  April 12, 2010
  I can't help feeling that if this was worth doing it wouldn't be necessary. After 10 years or so trying to get Windows admins/devs to consider open source solutions my main experience is that the prevailing culture on Windows is not the 'do-it-yourself' mentality that saw the creation of the OSS you're aiming at but more a 'I'll-pay-someone-to-do-it-for-me' culture. Creating a nice point and click repository of Linux ports might help get that software onto more Windows machines but would do nothing to promote the open-source ideal and help foster a healthy open-software community on Windows. Frankly, if the future of open-source on Windows is nothing more than a collection of ill-fitting Linux ports then Windows open source is dead and we may as well admit it. Also, I'm a little puzzled at your choice of Apache and Python - these already have excellent Windows versions which install as easily as any Windows apps I've used. As for PHP given its threading issues and the need to restrict it to a FastCGI host I'd say this is a good example of why some Linux apps will never work as well on Windows as on Linux and the ultimate pointlessness of looking to port them for anything other than development use.

• Anonymous
  April 12, 2010
  The comment has been removed

• Anonymous
  April 12, 2010
  For Python on windows, the closest to Right Thing distro currently is http://pythonxy.com - they provide independent modules as seperate installers (though EXE and not MSI), but you must install dependencies manually; if you opt to use their big install, you also get all dependencies and updates automatically. I guess they'd be happy to use CoApp instead for dependencies. Another projects that might make a lot of sense for early conversion to CoApp is Cygwin, which also currently uses its own dependency-managing installer. Because like it or not, software originally developed for linux will have a unixish build process, and Cygwin/Mingw provide the easiest path to replicate such build on windows.